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ASSESSING  THREAT  DETECTION  SCENARIOS  THROUGH  HYPOTHESIS 
GENERATION  AND  TESTING 

EXECUTIVE  SUMMARY 


Research  Requirement: 

The  operational  environment  often  introduces  severe  time  limitations  and  high 
uncertainty,  requiring  Soldiers  to  make  immediate  decisions  about  potential  threats  in  high-risk 
and  high-stakes  situations.  The  overarching  goal  of  this  research  was  to  explore  the  decision¬ 
making  processes  of  Soldiers  with  different  levels  of  experience  as  they  evaluated  scenarios  of 
varying  levels  of  uncertainty.  This  research  was  particularly  focused  on  understanding  the 
interaction  of  experience  and  uncertainty  on  hypothesis  generation  and  testing  and  on  the 
relationship  between  confidence  and  decision-making.  In  order  to  achieve  the  research  focus, 
the  experiment  we  conducted  allowed  us  to  clarify  how  Soldiers  generate  and  test  hypotheses  by 
selecting  and  integrating  cues  in  uncertain  decision  environments  involving  potential  threats. 
Specific  research  questions  investigated  whether  experienced  and  inexperienced  Soldiers 
differed  in  the  priority  threats  they  identified  in  a  given  scenario,  their  confidence  in  their  initial 
assessments,  the  number  and  type  of  cues  they  used  to  make  judgments,  and  the  adjustments 
made  to  their  hypotheses  after  seeking  additional  information. 

Procedure: 

Fifty-nine  Soldiers  engaged  in  computer-based  exercises  that  measured  decision-making 
performance  in  a  threat  detection  task.  The  exercises  involved  reading  threat-relevant  scenarios 
and  then  reporting  threat  decisions.  Each  Soldier  read  two  certain  and  two  uncertain  scenarios. 
After  reading  each  scenario,  Soldiers  performed  a  threat  assessment  by  identifying  the  highest 
priority  threat  and  explaining  the  importance  of  this  threat.  They  then  rated  their  confidence  in 
their  initial  assessments  and  described  the  cues  and  information  they  used  to  make  their 
decisions.  After  each  threat  assessment,  Soldiers  described  their  course  of  action  (COA)  and 
provided  a  confidence  rating  for  how  successful  this  COA  would  be  at  mitigating  the  identified 
threat.  Soldiers  could  then  gather  more  information  about  each  scenario  from  six  topics.  After 
reviewing  additional  information,  Soldiers  had  the  opportunity  to  revise  their  original  threat 
assessment.  They  also  provided  alternative  explanations  for  each  scenario  and  alternative  action 
choices. 

Findings: 

Experienced  and  inexperienced  Soldiers  focused  on  different  priority  threats  across  three 
of  the  four  scenarios,  though  their  overall  patterns  of  responding  did  not  differ.  That  might 
indicate  that  Soldiers  who  previously  deployed,  and  went  outside  the  wire,  attend  to  different 
cues  in  the  environment  and  possibly  process  these  cues  differently.  Experienced  Soldiers  were 
likely  to  report  more  discrete  threats  when  identifying  their  priority  threat  in  each  scenario.  They 
were  also  more  likely  to  search  information  that  confirmed  their  initial  hypotheses.  Both 
experienced  and  inexperienced  Soldiers  tended  to  report  relatively  high  confidence  in  their  initial 
threat  decisions.  Changes  in  hypotheses  appeared  to  be  associated  with  lower  initial  confidence 
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ratings.  That  indicates  that  Soldiers  may  have  been  relatively  well  calibrated.  Experienced  and 
inexperienced  Soldiers  did  not  differ  on  other  measures,  such  as  the  variability  in  their  priority 
threats,  the  number  of  cues  identified  as  informative,  and  whether  they  searched  relevant 
additional  information  topics.  Perhaps  these  results  reflect  the  constraint  imposed  by  the  testing 
materials,  such  as  a  limited  number  of  cues  available  to  inform  the  initial  hypothesis;  however, 
they  may  also  reflect  general  trends  in  cognition.  Across  experience  levels,  Soldiers  tended  to 
search  relevant  details  more  often  than  irrelevant  details. 

Utilization  and  Dissemination  of  Findings: 

Overall,  this  experiment  supports  the  idea  that  differences  exist  between  novice 
(inexperienced)  and  expert  (experienced)  Soldiers’  decision-making  processes,  particularly  in 
operationally  relevant  domains.  The  finding  that  Soldier’s  initial  confidence  seemed  to  be 
associated  with  their  inclination  to  change  their  hypotheses  is  promising,  particularly  if  they  are 
inclined  to  evaluate  and  revise  hypotheses  when  they  are  not  maximally  confident.  Whether  or 
not  Soldiers  changed  their  initial  hypotheses  is  a  reasonable  measure  of  how  plausible  they 
thought  their  hypotheses  were,  even  after  the  opportunity  to  evaluate  them  critically.  Across 
scenarios,  there  was  little  difference  between  experienced  and  inexperienced  Soldiers  in 
identifying  informative  cues.  That  finding  comports  with  literature  that  suggests  decision 
makers  generate  hypotheses  using  relatively  small  sets  of  data  (e.g.,  Klein,  1998).  Future 
research  should  conduct  testing  in  naturalistic  environments  to  gain  a  greater  understanding  of 
Soldier  decision-making  processes.  Using  a  more  naturalistic  environment  might  mitigate 
limitations  of  memory  inherent  in  computer-based  tests,  where  participants  only  passively 
experience  the  scenario  through  text  rather  than  a  comprehensive  experience  (e.g.,  via  live 
scenarios). 
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ASSESSING  THREAT  DETECTION  SCENARIOS  THROUGH 
HYPOTHESIS  GENERATION  AND  TESTING 


Decision  makers  must  engage  in  several  cognitive  processes  prior  to  taking  action, 
including  perceiving  and  processing  situational  cues  and  then  generating,  evaluating  (or 
implicitly  testing),  and  revising  hypotheses  as  necessary.  These  processes  rely  on  a  complex 
interaction  of  factors,  including  background  knowledge,  situational  cues,  environmental  factors, 
and  the  cognitive  resources  necessary  to  perceive  and  evaluate  this  incoming  information.  A 
critical  decision  Soldiers  must  often  make  in  operational  environments  is  whether  (and  to  what 
extent)  a  situation  poses  a  threat.  That  decision  guides  how  Soldiers  determine  and  implement 
courses  of  action.  When  evaluated  collectively,  cues  provide  a  holistic  representation  of  events 
that  guide  how  decision  makers  perceive,  interpret,  and  act  upon  situations.  Soldiers  attempting 
to  detect  threats  and  make  what  are  sometimes  life-saving  decisions,  must  operate  in 
environments  that  present  ambiguous,  missing,  or  continually  changing  threat  cues.  These 
instances  can  lead  to  uncertainty  about  the  situation  and  the  action  to  take. 

Uncertainty  forces  decision  makers  to  either  delay  a  decision  in  an  effort  to  reduce 
uncertainty  or  forego  uncertainty-reducing  processes  and  make  an  immediate  decision  based  on 
the  available  information  (even  if  incomplete  and/or  biased).  Both  actions  have  advantages  and 
disadvantages,  and  they  will  be  more  or  less  likely  to  occur,  depending  upon  the  cost  of  delaying 
a  decision  (Cohen,  Freeman,  &  Thompson,  1998).  If  the  cost  of  delaying  a  decision  is 
unacceptable  (e.g.,  the  severity  of  a  potential  threat  increases  over  time),  the  decision  maker 
makes  an  immediate  decision,  even  if  the  outcome  is  suboptimal.  If  the  cost  of  delaying  a 
decision  is  acceptable,  the  decision  maker  will  attempt  to  reduce  uncertainty  by  seeking  more 
information  and  increasing  the  odds  of  making  an  optimal  decision.  The  operational 
environment  often  introduces  severe  limitations  on  time,  requiring  Soldiers  to  make  immediate 
decisions  about  potential  threats  even  in  the  face  of  uncertainty.  Making  decisions  in  these  high- 
risk,  and  high-stakes,  situations  requires  complex  cognitive  processing.  A  goal  of  this  research 
was  to  elucidate  some  of  the  judgment  and  decision-making  processes  (i.e.,  hypothesis 
generation  and  evaluation)  inherent  in  conditions  marked  by  uncertainty,  specifically  when 
detecting  threats. 

Osman  (2010)  reported  that  novices  and  experts  differed  in  how  they  processed 
information  in  uncertain  environments  (see  also  Lipshitz  &  Strauss,  1997).  Experienced  Soldiers 
may  quickly  recognize  and  interpret  critical  cues  that  help  them  interpret  the  decision 
environment  and  formulate  hypotheses,  whereas  inexperienced  Soldiers  may  need  to  seek  more 
information  before  developing  a  hypothesis  and  then  making  sense  of  the  situation.  In  addition, 
experienced  decision  makers  tend  to  take  into  account  novel  information  and  adjust  their 
assessments  to  account  for  this  information.  Experienced  decision  makers  tend  to  be  more  fluid 
than  novices  are  when  adapting  their  current  assessments  of  the  situation  to  account  for  novel  or 
anomalous  information  (Dreyfus  &  Dreyfus,  1986;  Ross,  Phillips,  Klein,  &  Cohn,  2005). 

Another  goal  of  this  research  was  to  explore  Soldier  decision-making  at  different  levels  of 
experience  and  uncertainty.  We  predicted  that  Soldiers  with  more  experience  would  have  a 
greater  proficiency  for  detecting  and  reasoning  about  threats.  These  proficiencies  allow  decision 
makers  to  generate  and  test  hypotheses  about  uncertain  decision  environments. 
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The  following  research  explores  the  decision-making  processes  of  Soldiers  with  different 
levels  of  experience  as  they  evaluate  scenarios  with  differing  levels  of  uncertainty.  This 
experiment  focuses  on  understanding  the  interaction  of  experience  and  uncertainty  on  hypothesis 
generation  and  testing  and  on  the  relationship  between  confidence  and  decision  making.  As  a 
result,  we  attempted  to  gain  a  better  understanding  as  to  how  Soldiers  generate  and  test 
hypotheses  by  selecting  and  integrating  (informational)  cues  in  uncertain  decision  environments 
involving  potential  threats.  Specific  research  questions  investigated  whether  experienced  and 
inexperienced  Soldiers  differed  in  the: 

•  Priority  of  threats  they  identify  in  a  given  scenario; 

•  Confidence  they  have  in  their  initial  assessment  of  the  scenario; 

•  Number  of  cues  they  identify  to  support  their  initial  hypotheses  (threat  decisions); 

•  Extent  to  which  they  seek  information  that  confirms  their  initial  hypotheses;  and, 

o  Additional  information; 

o  Relevant  versus  irrelevant  information;  and, 

•  Adjustments  to  their  hypotheses  after  seeking  additional  information. 

In  the  current  experiment,  we  manipulated  the  certainty  of  decision  contexts  to  measure 
the  extent  to  which  Soldiers  of  varying  experience  levels  appear  to  engage  in  verbatim  versus 
gist  processing  across  different  threat  situations.  We  examined  the  number  of  cues  used  by  both 
experienced  and  inexperienced  Soldiers  across  different  conditions  of  uncertainty  and  expected 
that  experienced  Soldiers  would  identify  fewer  cues  relative  to  inexperienced  Soldiers  and  that 
they  would  form  a  more  comprehensive  view  of  the  environment. 

Hypothesis  Generation 

When  decision  makers  generate  hypotheses,  they  create  potential  explanations  given  a  set 
of  information  (Fisher,  Gettys,  Manning,  Mehle,  &  Baca,  1983;  Gettys  &  Fisher,  1979).  Across 
a  variety  of  decision  contexts,  decision  makers  tend  to  generate  hypotheses  naturally  (Thomas, 
Dougherty,  Sprenger,  &  Harbison,  2008).  Hypothesis  generation  typically  includes  creating 
explanations  that  inform  the  evaluation  and  testing  of  any  one  potential  solution  or  course  of 
action.  For  example,  in  recognition-based  decision  models  (e.g.,  the  Recognition-Primed 
Decision  model;  Klein,  1997),  experienced  decision  makers  identify  critical  cues  in  the 
environment  and  match  these  cues  to  details  in  memory  of  previously  experienced  situations. 
From  this  match,  decision  makers  can  quickly  recognize  the  situation,  develop  a  hypothesis,  and 
predict  an  outcome.  Research  tends  to  show  that  expert  decision-making  in  uncertain 
environments  relies  heavily  on  pattern  matching.  Cohen,  Freeman,  and  Wolf  (1996)  reported 
that  features  of  the  decision  problem  activates  related  schemas  through  recognitional  pattern 
matching.  As  decision  makers  gain  experience,  more  patterns  are  stored  in  memory,  which 
enables  decision  makers  to  more  quickly  recognize  and  respond  to  situations  (see  also  Klein, 
1993;  Klein,  1997;  Fipshitz  &  Strauss,  1997).  However,  when  the  environmental  cues  do  not 
match  the  event  features  in  memory,  recognition  does  not  occur  immediately.  In  those  cases, 
experienced  decision  makers  will  generate  alternative  hypotheses  in  an  attempt  to  achieve  a 
recognitional  match  to  existing  schemas.  In  contrast,  less  experienced  decision  makers  lack  a 
comprehensive  store  of  relevant  events  in  memory  and  thus  have  difficulty  matching 
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environmental  cues.  They  instead  generate  multiple  possibilities  informed  by  a  wider  range  of 
available  cues. 

The  observed  differences  between  expert  and  novice  hypothesis  generation  processes 
may  be  due  to  the  nature  of  the  decisions  examined  (or  the  tasks  performed;  e.g.,  see  Norman, 
Brooks,  &  Allen,  1989).  Norman  et  al.  contrasted  the  task  of  playing  chess  with  that  of 
diagnosing  medical  conditions.  Whereas  both  chess  players  and  physicians  must  recognize 
patterns  among  a  fixed  set  of  cues,  physicians  in  high-stakes  environments  must  also  be  able  to 
evaluate  a  potentially  unlimited  number  of  novel  cues  and  search  for  additional  cues,  all  while 
maintaining  these  details  in  memory.  Thus,  the  common  research  finding  that  chess  experts  can 
hold  and  process  more  information  chunks  in  memory  than  can  novices  (e.g.,  Chase  &  Simon, 
1973)  does  not  necessarily  generalize  to  expert  physicians  (e.g.,  see  Patel,  Arocha,  &  Kaufman, 
1994).  Similar  to  the  demanding  task  of  medical  diagnosis,  detecting  threats  is  highly  complex 
and,  when  embedded  in  dynamic  contexts,  likely  requires  decision  processes  that  change  along 
with  the  situation.  For  example,  a  situation  that  matches  previous  experience  and  allows  for 
some  certainty  will  likely  provoke  a  quick  hypothesis  generation  process.  By  contrast,  a 
situation  that  fails  to  match  previous  experience  and  causes  uncertainty,  will  likely  lead  to  a 
slower,  more  deliberate  hypothesis  generation  process. 

Regardless  of  the  model  or  theory  enlisted  to  explain  hypothesis  generation  (e.g.,  see 
Cohen  et  al.,  1998;  Klein,  1998;  Reyna,  Lloyd,  &  Brainerd,  2003)  the  role  of  memory  appears  to 
be  a  critical  component  (e.g.,  see  Gettys,  Manning,  Mehle,  &  Fisher,  1980).  Details  in  the 
decision  environment  serve  as  retrieval  cues  that  likely  differ  across  contexts.  It  is  unclear  how 
many  cues  decision  makers  typically  use  to  construct  hypotheses,  though  some  researchers  argue 
that  experienced  decision  makers  use  only  a  small  number  of  cues  to  produce  decisions  (e.g., 
Klein,  1998).  Regardless  of  the  number  of  cues  evaluated,  research  also  shows  that  experts  focus 
on  more  informative  or  applicable  cues  relative  to  novices  (Shanteau,  1992). 

One  theory  that  explains  differences  in  the  number  of  cues  decision  makers  use  to  form  a 
hypothesis  is  the  fuzzy  trace  theory  (Reyna  et  al.,  2003).  This  theory  assumes  that  two  forms  of 
processing  work  to  form  a  hypothesis:  verbatim  and  gist  processing.  Verbatim  processing 
entails  evaluating  all  of  the  cues  in  a  decision  environment.  For  example,  a  Soldier  may  attend 
to  the  area  of  interest,  for  instance,  an  area  of  disturbed  earth.  They  will  also  evaluate  the 
undisturbed  earth  around  the  area,  a  nearby  wall,  vehicles  in  the  immediate  vicinity,  the  distance 
to  the  nearest  dwelling,  the  time  of  day,  fresh  tire  tracks,  and  other  cues  to  generate  an 
understanding  of  the  environment  and  to  determine  the  threat  relevance  of  the  disturbed  earth. 
Reyna  and  colleagues  suggested  that  novices  use  a  large  number  of  details,  such  as  these,  to 
generate  a  hypothesis  about  the  decision  environment.  By  contrast,  experts  tend  to  engage  in  gist 
processing,  in  which  they  attend  to  relatively  few,  but  critical  cues  to  generate  a  holistic 
representation  of  the  decision  environment  (see  also  Werner  and  Thies,  2000,  for  a  discussion  of 
domain- specific  influence  on  cue  utilization).  For  example,  to  generate  a  rough  understanding  of 
the  environment  and  its  threat  relevance,  an  experienced  Soldier  may  attend  only  to  the  area  of 
disturbed  earth,  the  distance  to  the  nearest  dwelling,  and  local  traffic  patterns. 
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Hypothesis  Testing 

When  decision  makers  engage  in  hypothesis  testing,  they  check  the  explanations  they 
formed  to  account  for  the  decision  environment.  As  with  hypothesis  generation,  multiple 
theories  and  models  exist  to  characterize  and  explain  hypothesis  testing  (e.g.,  Cohen  et  al.,  1996; 
Klein,  Moon,  &  Hoffman,  2006;  Osman,  2010).  Unlike  hypothesis  generation,  however, 
researchers  often  consider  testing  to  be  a  deliberative  cognitive  process.  For  example,  Cohen  et 
al.  suggested  that  decision  makers  engage  a  set  of  serial  processes  to  evaluate  the  accuracy  of 
their  hypotheses  under  conditions  of  uncertainty.  They  evaluate  their  hypotheses  to  determine 
completeness  (Is  there  sufficient  support  for  the  decision?),  potential  conflict  (Are  two 
incompatible  outcomes  supported?),  and  reliability  (Is  the  cue-to-outcome  relationship  supported 
by  examined  vs.  unexamined  assumptions?).  If  their  testing  reveals  flaws,  decision  makers  will 
attempt  to  correct  the  hypothesis  by  gathering  more  information  or  re-considering  information. 
When  additional  information  corrects  the  flaw,  for  instance  by  filling  in  gaps  in  knowledge, 
resolving  cue-outcome  conflicts,  or  confirming/denying  assumptions,  the  decision  maker  adjusts 
the  hypothesis  and  re-tests  it. 

Under  conditions  of  certainty,  initial  hypotheses  tend  to  be  complete,  compatible  with 
outcomes,  and  reliable.  Further  evaluation  and  information  seeking  is  unnecessary  and 
inefficient  in  such  cases,  thus  decision  makers  take  action  based  on  the  expected  outcomes  of  the 
hypotheses.  Cohen  et  al.  (1996)  recognized  that  the  perceived  cost  of  hypothesis  testing 
influences  the  willingness  of  decision  makers  to  engage  in  deliberative  processing.  If  the  cost  of 
hypothesis  testing  appears  too  high,  or  the  resources  required  for  testing  are  unavailable,  then  a 
decision  may  occur  quickly,  but  at  the  risk  of  being  only  partially  informed  and  unreliable.  The 
operational  environment  may  regularly  produce  situations  in  which  the  costs  of  effortful 
hypothesis  testing  are  too  high,  and  thus  decision  makers  must  make  quick  decisions  without 
additional  testing.  In  the  present  experiment,  we  examined  the  influence  of  uncertainty  and 
experience  on  hypothesis  generation  and  testing  when  the  cost  of  testing  was  salient. 

Specifically,  we  assigned  costs  to  each  information  search  that  Soldiers  engaged  in  after  they 
formulated  their  initial  hypothesis.  Because  experienced  decision  makers  can  draw  on  previous 
experiences,  there  are  lower  risks  associated  with  taking  action  without  further  testing  and  a 
reduced  sense  of  uncertainty.  Thus,  we  expected  that  in  conditions  of  uncertainty  experienced 
Soldiers  would  seek  fewer  pieces  of  additional  information  compared  to  inexperienced  Soldiers. 
Conversely,  under  conditions  of  certainty,  experienced  and  inexperienced  Soldiers  would  seek 
additional  information  at  the  same  rate. 

In  addition  to  examining  the  amount  of  information  Soldiers  gathered  to  test  hypotheses, 
we  compared  the  type  of  information  experienced  and  inexperienced  Soldiers  chose  to  consider 
when  testing  their  hypotheses.  Shanteau  (1992)  suggested  that  expert  decision  makers  are  better 
able  to  differentiate  between  relevant  and  irrelevant  information.  However,  he  identified  several 
domains  in  which  expert  decision  makers  performed  no  better  than  did  novice  decision  makers. 
Further,  Raab  and  Johnson  (2007)  noted  differences  in  the  generation  of  options  and  strategy 
selection  based  on  level  of  expertise.  The  current  experiment  examined  the  influence  of 
uncertainty  and  experience  on  the  relevance  of  the  information  Soldiers  sought  to  test 
hypotheses.  We  forced  decision  makers  to  choose  at  least  one  piece  of  additional  information 
from  a  set  of  information  topics  that  included  both  relevant  and  irrelevant  details,  after  they  had 
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indicated  their  initial  hypotheses.  We  expected  that  experienced  Soldiers  would  choose  relevant 
details  more  often  than  irrelevant  details  and  inexperienced  Soldiers  would  choose  relevant  and 
irrelevant  details  equally.  We  also  predicted  an  interaction  between  experience  and  uncertainty. 
We  expected  that  in  the  uncertain  condition,  experienced  Soldiers  and  inexperienced  Soldiers 
would  differ  in  selecting  relevant  versus  irrelevant  details,  whereas  under  conditions  of  certainty, 
experienced  Soldiers  and  inexperienced  Soldiers  would  not  differ  in  selecting  relevant  details. 

Forcing  decision  makers  to  choose  at  least  one  piece  of  additional  information  allowed  us 
to  explore  the  extent  to  which  Soldiers  demonstrate  confirmation  bias  by  choosing  information 
that  served  to  confirm  their  initial  hypothesis.  Confirmation  bias  is  one  of  the  most  common 
decision  biases,  in  which  decision  makers  search  for  information  or  make  interpretations  that 
confirm  their  pre-existing  beliefs.  By  doing  this,  they  may  sometimes  ignore  or  dismiss  other 
important  information,  leading  to  errors  in  judgment.  Researchers  have  generally  found  that 
decision  makers  prefer  to  take  steps  toward  confirming  rather  than  disconfirming  their 
hypotheses.  For  example,  Adsit  and  London  (1997)  found  that  participants  generally  sought  to 
confirm  the  hypotheses  they  generated.  When  experimenters  provided  the  hypotheses, 
participants  produced  more  disconfirmations  than  they  did  when  they  generated  their  own 
hypotheses  or  when  they  received  hypotheses  generated  by  other  participants.  Soldiers  in  the 
present  experiment  generated  and  tested  their  own  hypotheses,  thus  we  expected  that  they  would 
seek  out  information  to  confirm  their  hypotheses.  Furthermore,  we  expected  that  displays  of 
confirmation  bias  would  be  associated  with  greater  confidence  in  their  initial  hypothesis. 

In  addition  to  confirmation  bias,  measures  of  confidence  ordinarily  allow  for 
examinations  of  calibration  (for  a  review,  see  Griffin  &  Brenner,  2008).  Typically,  subjective 
probabilities  of  accuracy  (confidence)  that  align  with  similar  objective  probabilities  of  accuracy 
(actual  accuracy)  indicate  good  calibration.  By  contrast,  misaligned  probabilities  indicate  poor 
calibration.  In  this  experiment,  there  were  no  objective  measures  of  accuracy  because  no  real 
(experienced)  threats  existed  in  the  scenarios  (thus,  there  was  no  objective,  correct  answer).  To 
assess  calibration  in  this  context,  we  measured  the  association  between  confidence  and  accuracy 
by  examining  changes  in  hypotheses  at  different  levels  of  initial  confidence.1  If  high-confidence 
hypotheses  change  over  time,  then  we  might  reasonably  assume  that  calibration  was  less  than 
optimal.  By  contrast,  changes  in  low-confidence  hypotheses  likely  reflect  better  calibration. 

Experiment 

Method 

Design.  Experienced  and  inexperienced  Soldiers  each  completed  four  decision-making 
scenarios,  each  of  which  included  the  same  set  of  eight  questions  (see  Appendix  A).  The  level  of 
threat  varied  across  the  scenarios  with  two  scenarios  presenting  relatively  clear,  unambiguous 
threats  and  two  scenarios  presenting  ambiguous  threats  intended  to  increase  uncertainty. 
Presented  along  with  each  scenario  were  six  additional,  searchable  information  topics  that  were 
either  relevant  to  the  threat  scenario  or  irrelevant.  Soldiers  were  required  to  search  one  topic  and 
could  search  up  to  six  topics  per  scenario.  This  experiment  used  a  mixed  design,  with  contextual 


1  Unchanged  hypotheses  would  be  difficult  to  interpret  in  this  context,  as  despite  being  stable  they  may  also  be 
inaccurate. 


5 


cues  (certain  vs.  uncertain)  as  a  within-subjects  factor  and  Soldier  experience  as  a  between- 
subjects  fixed  factor.  Dependent  measures  were: 


•  Initial  threat  decision  (i.e.,  identification  of  the  highest  priority  threat); 

•  Confidence  in  initial  threat  decision; 

•  The  number  of  cues  that  informed  the  initial  threat  decision; 

•  The  number  of  searchable  information  topics  explored; 

•  The  proportion  of  relevant  to  irrelevant  details  explored;  and, 

•  Changes  to  the  initial  threat  decision. 

Participants.  Fifty-nine  Soldiers  participated  in  this  research.  Their  mean  age  was  23 
years  (SD  =  4.7).  These  Soldiers  had  paygrades  ranging  from  E-l  to  E-6  (see  Table  1).  Their 
mean  time-in-service  was  2.8  years  (SD  =  3.0).  Thirty-four  Soldiers  (58%)  deployed  previously. 
We  categorized  those  Soldiers  as  experienced  as  they  had  increased  likelihood  of  carrying  out 
threat  detection  on  actual  patrols  as  opposed  to  those  who  had  not  deployed.  To  clarify  the 
‘experienced’  categorization,  we  further  asked  deployed  Soldiers  how  often  they  went  “outside 
the  wire.”  Of  the  Soldiers  who  previously  deployed,  6%  reported  never  going  outside  the  wire, 
3%  reported  going  outside  the  wire  less  than  once  per  month,  3%  reported  going  outside  the  wire 
once  per  month,  85%  reported  going  outside  the  wire  more  than  once  per  month,  and  3% 
reported  going  outside  the  wire  once  per  day.2 

Experience  Grouping  Variable.  We  operationalized  experience  (experienced  vs. 
inexperienced)  based  on  deployments.  Experienced  Soldiers  had  at  least  one  prior  deployment 
while  inexperienced  Soldiers  had  zero  deployments.  Thirty-five  Soldiers  reported  at  least  one 
deployment.  Of  these,  24  were  deployed  once,  and  1 1  were  deployed  more  than  once.  The 
maximum  number  of  deployments  reported  was  five  ( M  =  1.46,  SD  =  .85). 

Scenarios.  Scenarios  were  developed  based  on  contemporary  experiences  of  active  duty 
Soldiers  and  follow  from  data  collected  from  previous  research  completed  involving  Soldiers’ 
experiences  conducting  threat  detection  in  operational  settings  (Zimmerman,  Mueller,  Grover,  & 
Vowels,  2014).  Each  scenario  included  a  short  description  of  a  military  tactical  threat  situation. 
Two  scenarios  provided  concrete  information  about  potential  threats  (e.g.,  a  statement  indicating 
that  bomb  sniffing  dogs  kept  returning  to  a  specific  location);  this  allowed  for  more  certainty  in 
assessment  and  decisions  (see  “Dog  Day”  and  “The  Rock”  in  Appendix  A).  Two  scenarios 
presented  more  uncertainty  by  providing  only  ambiguous  information  about  potential  threats 
(e.g.,  a  statement  indicating  that  some  local  nationals  are  interested  in  inciting  violence,  but  that 
local  elders  do  not  condone  it;  see  “Checkpoint”  and  “Election”  in  Appendix  A).  After  reading 
each  scenario,  Soldiers  answered  two  questions  (each  with  two  sub-questions)  about  the  potential 
threats  in  the  scenario.  Following  these  questions,  Soldiers  could  search  additional  information 
to  gain  a  better  understanding  of  the  situation  (see  the  bolded  labels  on  pg  A-2).  They  could 
select  information  by  choosing  from  among  six  searchable  topics,  three  of  which  were  relevant 
to  potential  threats  in  the  scenario  and  three  of  which  were  irrelevant  to  potential  threats. 

Relevant  searchable  topics  contained  information  that  could  assist  in  identifying  a  priority  threat, 


2  We  specifically  examined  the  12%  of  Soldiers  who  reported  going  outside  the  wire  once  per  month  or  less  and 
found  no  statistically  significant  difference  in  our  analyses  whether  we  included  them  in  the  “inexperienced”  or 
“experienced”  groups. 


6 


whereas  irrelevant  topics  provided  information  that  was  either  arbitrary  or  did  not  offer  useful 
intelligence.  After  searching  through  these  topics,  participants  could  review  and  revise  their 
original  assessment.  Finally,  they  finished  each  scenario  by  answering  two  more  questions  about 
possible  alternatives  to  their  previous  responses  (see  Appendix  A  for  a  full  description  of  each 
scenario  and  the  questions). 

Table  1 

Soldier  Rank,  Number  Deployed,  and  Military  Occupational  Specialty  (MOS) 


Soldiers  with  Zero  Deployments  ( n  =  25) 

MOS 

Rank 

Total 

1  IB 

19K 

68W  91F 

92F 

Other 

Private 

1 

1 

PV2 

10 

4 

3 

1  1 

1 

PFC 

12 

5 

3 

1  1 

2 

CPL/SPC 

2 

1  1 

SGT 

0 

SSG 

0 

Soldiers  with  One  or  More  Deployments  (n  =  34) 

MOS 

Rank 

Total 

1  IB 

19K 

68W  91F 

92F 

Other 

Private 

0 

PV2 

0 

PFC 

6 

3 

2 

1 

CPL/SPC 

19 

5 

5 

1 

2 

6 

SGT 

8 

3 

2 

1 

1 

1 

SSG 

1 

1 

Note:  MOS  are  1  IB  =  Infantryman,  19K  -  Ml  Armor  Crewman,  68W  =  Combat  Medic,  91F  =  Small 
Arms/ Artillery  Repairer,  92F  =  Quartermaster  Corps.  Other  includes  19D,  29E,  35F,  88M,  92A,  and 
92Y. 


Manipulation  Check.  To  ensure  that  the  scenarios  differed  in  the  amount  of  certainty  or 
uncertainty  they  afforded  decision  makers,  they  were  evaluated  by  a  small  sample  of  experienced 
Soldiers  ( n  =  5)  and  revised  according  to  their  feedback.  Furthermore,  during  debriefing,  a 
random  subset  of  participants  ( n  =  37)  identified  the  scenario  or  scenarios  they  found  to  be 
notable  and  most  difficult  (vs.  easy).  Sixty-nine  percent  of  the  Soldiers  who  identified  the 
scenarios  with  high  certainty  as  notable  (“Dog  Day”  and  “The  Rock”)  indicated  that  those 
scenarios  were  the  easiest.  Sixty-seven  percent  of  the  Soldiers  who  identified  the  scenarios  with 
high  uncertainty  as  notable  (“Checkpoint”  and  “Election”)  indicated  that  those  scenarios  were 
the  most  difficult.  Those  differences  were  statistically  significant,  y2  (1,  37)  =  4.56,  p  =  .0333. 

To  check  the  relevance  of  the  searchable  information,  we  examined  the  rates  at  which 
Soldiers  searched/chose  each  type  of  information  topic.  Relevant  details  were  searched 


3This  measure  is  indirect  and  subjective  and,  as  such,  should  be  used  cautiously  in  subsequent  analyses.  For  this 
reason,  we  did  not  collapse  the  scenarios  into  two  groups  (certain  vs.  uncertain)  and  instead  kept  them  separated  in 
subsequent  analyses. 
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significantly  more  often  than  were  irrelevant  details  ( M  =  .80,  SI)  =  .45  vs.  M  =  .50,  SI)  =  .38), 
t(54)  =  3.95,  p  <  .001,  d  =  .82. 

Computer-controlled  stimuli  presentation.  The  Psychology  Experiment  Building 
Language  (PEBL;  Mueller,  2009)  program  was  used  to  control  the  computer  exercises.  Soldiers 
completed  the  experiment  by  viewing  and  responding  to  all  stimuli  and  questions  in  this 
program.  PEBL  recorded  all  responses  for  later  analysis. 

Demographic  questionnaire.  Soldiers  completed  a  demographic  questionnaire  as  part  of 
the  computerized  tasks.  The  questionnaire  included  questions  about  education,  military 
experience,  threat  detection  training,  and  specific  field  experience  relevant  to  threat  detection 
(Appendix  B). 

Procedure.  Soldiers  engaged  in  computer-based  exercises  that  measured  decision¬ 
making  performance  in  a  threat  detection  task.  These  exercises  involved  reading  threat-relevant 
scenarios  and  then  reporting  threat  decisions.  Each  Soldier  read  four  scenarios  (two  certain  and 
two  uncertain).  To  account  for  any  order  effects,  we  counterbalanced  the  scenarios  by 
programming  PEBL  to  randomize  the  presentation  order;  no  effects  of  order  where  found  (p 
>.05).  Soldiers  completed  a  brief  filler  task  (a  paper-based  word-find  exercise)  between 
scenarios  to  reduce  potential  interference  from  the  previous  scenario. 

Soldiers  who  participated  in  the  experiment  signed  informed  consent  forms,  completed 
the  demographic  questionnaire,  and  began  the  scenario  portion  of  the  experiment.  Soldiers  were 
informed  that  they  would  read  and  listen  (via  headphones)  to  10  threat  scenarios  on  a  computer. 
Before  the  Soldiers  began  the  scenarios,  a  researcher  gave  them  each  a  pencil  and  a  notepad  and 
told  them  that  they  could  take  notes  during  the  scenarios,  noting  that  the  Soldier  would  not  see 
the  scenario  details  when  it  was  time  to  respond  to  questions.  The  Soldiers  received  written 
instructions  that  following  each  scenario  they  would  have  the  opportunity  to  search  for 
information  about  that  scenario;  however,  each  information  search  would  cost  them  virtual 
money.  They  were  informed  that  they  would  begin  with  $200  and  needed  to  ensure  that  they  had 
enough  money  to  complete  all  10  scenarios  (i.e.,  they  should  not  spend  all  $200  on  the  first  nine 
or  fewer  scenarios).  Soldiers  completed  only  four  scenarios;  to  control  for  unnecessary  search 
(spending  too  much  of  their  resource  too  early)  and  to  emphasize  the  importance  of  search, 
instructions  indicated  they  would  complete  ten  scenarios.  Soldiers  were  debriefed  as  to  the 
purpose  of  that  manipulation  at  the  end  of  the  experiment.  They  were  also  informed  that  some 
scenarios  would  be  more  difficult  than  others,  thus  they  should  search  through  the  additional 
information  judiciously.  After  reading  each  scenario,  Soldiers  performed  a  threat  assessment, 
identifying  what  they  believed  was  the  highest  priority  threat  and  explaining  the  importance  of 
this  threat.  They  then  rated  their  confidence  in  their  initial  assessments  and  described  the  cues 
and  information  they  used  to  make  their  decisions. 

After  each  threat  assessment,  Soldiers  described  their  COA  and  provided  a  confidence 
rating  for  how  successful  this  COA  would  be  at  mitigating  the  identified  threat.  Soldiers  could 
then  gather  more  information  about  each  scenario  from  six  topics.  Each  information  search  cost 
$5  which  was  subtracted  from  their  initial  allocation  of  $200.  They  were  required  to  search  at 
least  one  topic,  but  had  the  option  of  selecting  up  to  six  pieces  of  information  per  scenario. 
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Soldiers  could  monitor  their  account  via  an  onscreen  account  indicator.  After  reviewing 
additional  information,  Soldiers  had  the  opportunity  to  revise  their  original  threat  assessment. 
They  also  provided  alternative  explanations  for  each  scenario  and  alternative  action  choices. 

Results  and  Discussion 

Hypothesis  generation:  Identifying  priority  threats.  We  analyzed  identified  threats 
using  Friedman’s  Rank  Order  test  (Elliott  &  Woodward,  2007),  which  ranks  items  chosen  most 
to  least  often.  The  results  were  graphed  by  scenario,  showing  rankings  by  experience  level  for 
each  identified  threat.  Magnitude  represents  the  relative  frequency  that  Soldiers  chose  an  item. 

In  addition  to  analyzing  the  rank  order  of  threats,  we  analyzed  the  overall  pattern  of  responses 
across  experience  level  using  Bhattacharyya’s  distance  equation  for  discrete  probability 
distributions  (e.g.,  see  Choi  &  Lee,  2003).  That  equation  quantifies  the  difference  between  two 
probability  distributions,  taking  into  account  all  possible  responses.  Distance  values  range  from 
0  to  1,  with  values  closer  to  0  indicating  dissimilarity  across  distributions,  and  values  closer  to  1 
indicating  similarity  across  distributions.  To  conduct  this  analysis,  priority  threat  data  were 
normalized  (the  mean  selection  probability  of  each  item  was  divided  by  the  mean  total  responses 
for  that  item)  to  compensate  for  unequal  sample  sizes  across  experience  groups. 

Dog  Day  scenario.  Both  groups  identified  improvised  explosive  devices  (IED)  most 
often  as  a  priority  threat  in  the  Dog  Day  scenario,  but  diverged  with  respect  to  the  perceived 
importance  of  other  threats;  for  example,  the  detainee’s  house  was  identified  as  a  priority  threat 
significantly  more  often  by  experienced  versus  inexperienced  Soldiers, x2  (1,  59)  =  5.98,  p  =  .014 
(Figure  1).  The  overall  pattern  of  response  between  the  two  groups  was  similar,  yielding  a 
Bhattacharyya  distance  of  .905. 


■  Inexperienced 

■  Experienced 


Priority  Threats 


Figure  1.  Rankings  of  priority  threats  identified  in  the  Dog  Day  scenario. 
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The  Rock  scenario.  Both  experienced  and  inexperienced  Soldiers  identified  spies  most 
often  as  a  priority  threat.  Experienced  and  inexperienced  Soldiers  differed  most  in  identifying 
the  leak  of  intelligence  to  potential  insurgents  (Intel)  as  a  priority  threat;  however,  this  difference 
did  not  reach  statistical  significance,  x2  (1,  59)  =  3.30,  p  =  .069  (Figure  2).  The  overall  pattern  of 
response  between  the  two  groups  was  similar,  yielding  a  Bhattacharyya  distance  of  .948. 


Priority  Threats 


■  Inexperienced 

■  Experienced 


Figure  2.  Rankings  of  priority  threats  identified  in  The  Rock  scenario. 


Checkpoint  scenario.  In  the  Checkpoint  scenario,  both  groups  identified  worsening 
relations  between  U.S.  Forces  and  the  host-nation  populace  as  the  priority  threat  relative  to  the 
other  threats.  There  was  also  agreement  between  the  groups  about  the  next  two  threats  in 
descending  rank  order:  young  local  nationals  and  a  breakdown  in  communications  between  U.S. 
leaders  and  local  elders  (Figure  3).  Only  one  threat  was  identified  differentially  by  experienced 
versus  inexperienced  Soldiers:  Experienced  Soldiers  were  more  likely  than  inexperienced 
Soldiers  to  identify  Intel  as  a  priority  threat,  j2  (1,  59)  =  4.02,  p  =  .045.  The  overall  pattern  of 
response  was  similar  between  experienced  and  inexperienced  Soldiers,  yielding  a  Bhattacharyya 
distance  of  .852. 
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■  Inexperienced 

■  Experienced 


Priority  Threats 


Figure  3.  Rankings  of  priority  threats  for  the  Checkpoint  scenario. 


Election  scenario.  In  the  Election  scenario,  inexperienced  and  experienced  Soldiers 
tended  to  select  the  vulnerability  of  the  election  headquarters  as  the  priority  threat.  There  was  no 
group  difference  in  the  frequency  with  which  this  threat  was  identified  as  the  priority,  yp  (1,  59) 

=  0.215,  p  =  .643.  Experienced  Soldiers  also  identified  as  a  priority  threat  the  uncertainty  they 
had  about  selecting  the  election  headquarters.  Compared  to  inexperienced  Soldiers,  experienced 
Soldiers  were  more  likely  to  identify  this  uncertainty  as  a  priority  threat,  x2  (1,  59)  =  4.25,  p  = 
.039.  A  related- samples  McNemar  test  revealed  that  experienced  Soldiers  selected  uncertainty 
and  the  election  headquarters  as  the  priority  threat  equally,  x2  (1,  34)  <  .001,  p  =  1.00  (Figure  4). 
The  overall  pattern  of  response  was  similar  between  experienced  and  inexperienced  Soldiers, 
yielding  a  Bhattacharyya  distance  of  .940. 
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Priority  Threats 


■  Inexperienced 

■  Experienced 


Figure  4.  Rankings  of  priority  threats  identified  in  the  Election  scenario. 


Cross-scenario  analysis.  Across  scenarios,  few  differences  in  threat  identification 
emerged  between  groups.4  We  did  not  observe  the  overall  pattern  of  responding  we  expected. 
Experienced  Soldiers  did  not  identify  smaller,  more  homogenous  sets  of  cues  than  did 
inexperienced  Soldiers.  Instead,  a  repeated-measures  analysis  of  variance  (ANOVA)  revealed 
that  experienced  Soldiers  tended  to  report  more  priority  threats  across  all  scenarios  ( M  =  6.32, 
SD  =  1.70)  than  did  inexperienced  Soldiers  (M  =  4.32,  SD  =  1.38),  F(l,  57)  =  4.83,  p  <  .01, 
partial  eta2  =  .291.  There  was  also  a  significant  main  effect  of  certainty  for  the  number  of 
threats  reported.  “Certain”  scenarios  (Dog  Day  and  The  Rock)  yielded  more  priority  threats  (M 
=  3.08,  SD  =  1.29)  than  did  “uncertain”  scenarios  (Checkpoint  and  Election;  M  =  2.38,  SD  = 
1.02),  F(l,  57)  =  1 2.76,  p  =  .001,  partial  eta2  =  .183,  but  there  was  no  significant  interaction 
between  levels  of  certainty  and  experience,  F(l,  57)  =  1.94,  p  =  .17,  partial  eta2  =  .033. 


One  explanation  for  why  the  certain  scenarios  yielded  more  priority  threat  identifications 
is  that  the  information  presented  in  those  scenarios  was  more  concrete,  thus  more  memorable 
over  time  (e.g.,  Walker  &  Hulme,  1999).  By  contrast,  the  uncertain  scenarios  offered  less  clarity 
and  concreteness;  thus,  the  Soldiers  may  have  forgotten  the  somewhat  ambiguous  threats  in  these 
scenarios  before  they  recorded  their  priority  threat  responses.  If  identifying  priority  threats  was 
in  part  contingent  upon  working  memory  (e.g.,  Thomas,  Dougherty,  Sprenger,  &  Harbison, 
2008),  then  we  would  expect  to  see  better  memory  performance  and  thus,  more  threats  identified, 
by  experienced  Soldiers,  who  might  have  an  increased  (or  more  efficient  use  of)  working 
memory  capacity  for  the  threat  detection  domain.  The  results  supporting  this  notion  showed  that 
experienced  Soldiers  did  identify  more  threats  than  did  inexperienced  Soldiers.  Overall, 
inexperienced  Soldiers  reported  more  than  one  threat  in  only  21%  of  scenarios,  whereas 


4  When  compared  to  a  conservatively  corrected  alpha  for  per-experiment  comparisons  (e.g.,  Bonferroni-corrected 
alphas  between  .005-. 006),  no  differences  across  groups  on  individual  threat  selection  reached  statistical 
significance,  no  p  <  .014. 
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experienced  Soldiers  reported  more  than  one  threat  in  45%  of  scenarios.  This  result  could 
indicate  that  experienced  Soldiers  possess  comprehensive  schemas,  increasing  their  ability  to 
recognize  and  recall  more  threat  possibilities  among  the  scenarios.  In  contrast,  inexperienced 
Soldiers  with  relatively  fewer,  less  developed  schemas  would  not  have  the  same  access  to 
memory  and  thus,  would  identify  fewer  threats.  We  would  also  expect  uncertain  scenarios  to 
have  little  impact  on  experienced  Soldiers’  memory,  thus  they  should  remember  similar  numbers 
of  threats  regardless  of  certainty  level.  However,  we  found  no  interaction  between  certainty  and 
experience,  indicating  that  both  groups  of  Soldiers  similarly  reported  fewer  threats  in  the 
uncertain  scenarios.  A  possible  explanation  for  this  finding  is  that  the  uncertain  scenarios  in  this 
experiment  were  sufficiently  ambiguous  and  difficult  enough  to  affect  both  experienced  and 
inexperienced  Soldiers  similarly. 

Hypothesis  generation:  Cues  identified  as  supporting  priority  threat  decision. 

Experienced  Soldiers  tended  to  identify  more  cues  across  scenarios  than  did  inexperienced 
Soldiers  (Table  2);  however,  this  difference  was  not  statistically  significant  across  all  scenarios, 
t( 57)  =  1 .58.  p  =  .12.  The  difference  between  experienced  and  inexperienced  Soldiers  reached 
statistical  significance  in  only  the  Election  scenario,  t(56.75)  =  2.21,  p  =  .03,  d  =  .54.  As  can  be 
seen  in  Table  2,  the  range  of  mean  differences  across  scenarios  is  quite  small.  It  is  unclear  what 
these  differences  might  mean  for  threat  decisions  in  the  real  world.  That  is,  the  effect  of 
considering  two  versus  three  critical  cues  when  making  a  decision  is  difficult  to  determine  from 
these  results.  These  results  do  support  the  notion  that  decision  makers  tend  to  rely  on  very  few 
cues  when  generating  their  hypotheses  (e.g.,  Klein  1998).  The  maximum  number  of  cues 
identified  in  any  scenario  was  five.  This  occurred  only  in  the  Dog  Day  scenario,  and  only  16% 
of  all  Soldiers  identified  more  than  three  cues  in  this  scenario.  In  all  other  scenarios,  the 
maximum  number  of  cues  identified  was  three,  with  fewer  than  20%  of  all  Soldiers  identifying 
three  cues  in  any  of  these  remaining  scenarios. 

Table  2 

Mean  Number  of  Cues  Identified  Per  Scenario 


Experience  Level 
Inexperienced  Experienced 


Scenario 

Mean 

SD 

Mean 

SD 

Mean  Difference 

P 

Effect  Size 

Dog  Day 

2.3 

1.27 

2.5 

1.01 

0.2 

.48 

d=Al 

The  Rock 

1.6 

0.57 

1.7 

0.76 

0.1 

.59 

d=  .15 

Checkpoint 

1.7 

0.67 

1.8 

0.74 

0.1 

.29 

d=.  14 

Election 

1.2 

0.44 

1.5 

0.06 

0.3 

.03 

d=.  54* 

Across  Scenarios 

1.7 

0.74 

1.9 

0.79 

0.175 

*  Denotes  an  effect  size  found  to  be  statistically  significant  at  alpha  =  .05;  however,  this  effect 
fails  to  meet  statistical  significance  at  the  Bonferroni-corrected  alpha  of  .0125  (correcting  for 
four  planned  comparisons ). 
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Examining  the  mean  number  of  cues  identified  within  levels  of  certainty  (i.e.,  “Dog  Day” 
and  “The  Rock”  vs.  “Checkpoint”  and  “Election”)  yielded  a  significant  main  effect  of  certainty. 
More  cues  were  identified  across  the  certain  scenarios  ( M  =  2.03,  SD  =  0.69)  than  across  the 
uncertain  scenarios  (. M  =  E58,  SD  =  0.50),  t( 58)  =  5.09,  p  <.01,  cl  =  .75.  There  was  no 
interaction  of  experience  and  scenario  on  the  number  of  cues  identified,  at  either  the  collapsed 
level,  F(l,  57)  =  0.31,  p  =  .58,  partial  eta2  =  .005,  or  treating  each  scenario  independently, 
Greenhouse-Geisser  corrected  F(2.43,  138.54)  =  .199,/?  =  .86 ,  partial  eta 2  =  .008. 

Experienced  and  inexperienced  Soldiers  performed  similarly  across  scenarios  perhaps 
because  the  scenarios  offered  a  finite  set  of  cues  to  which  they  could  attend.  In  contrast,  real- 
world  threat  decision  environments  likely  offer  nearly  countless  cues  that  can  inform  threat 
decisions.  Thus,  we  might  expect  to  find  a  greater  number  and  diversity  of  cues  identified  in 
more  naturalistic  decision-making  environments.  As  noted,  memory  appears  to  play  a  critical 
role  in  hypothesis  generation.  Perhaps  decision  makers  attend  only  to  what  they  can  maintain  in 
working  memory  (e.g.,  see  Thomas,  et  al.,  2008).  In  the  case  of  the  experimental  scenarios,  we 
removed  the  critical  information  and  the  scenario  context  before  Soldiers  identified  the  cues  they 
relied  on  to  choose  a  priority  threat.5  Similar  to  priority  threat  selection,  perhaps  the  number  of 
cues  identified  in  these  scenarios  was  a  function  of  Soldiers’  working  memory,  so  their 
recollection  of  cues  was  limited  by  what  they  could  retrieve  from  memory.  If  Soldiers  did  not 
attend  well  to  the  scenarios,  or  if  the  scenarios  presented  a  context  in  which  very  few  cues  were 
salient  or  memorable,  we  might  expect  to  find  few  cues  identified  as  informative.  This  may  have 
been  the  case  in  uncertain  scenarios,  particularly  the  Election  scenario,  which  yielded  lower 
numbers  of  identified  informative  cues  compared  to  certain  scenarios,  particularly  the  Dog  Day 
scenario. 

Interestingly,  the  Election  scenario  was  the  only  one  in  which  a  significant  difference  in 
identified  cues  existed  between  experienced  and  inexperienced  Soldiers.  This  scenario  also 
appears  to  have  caused  departures  from  trends  in  other  outcomes  as  well  (see  Hypothesis  testing, 
below).  This  scenario  generated  the  most  discussion  during  debriefing,  as  Soldiers  found  it  more 
remarkable  than  the  other  scenarios,  primarily  for  being  difficult.  This  scenario  included 
multiple  factors  ranging  from  abstract  (cultural)  to  concrete  (transport  logistics). 

Hypothesis  testing:  Seeking  additional  information.  There  was  no  main  effect  of 
scenario  on  the  number  of  additional  details  searched  by  scenario,  either  collapsed  within  levels 
of  certainty,  t( 58)  =  .130,  p  =  .90,  d  =  0.02,  or  treating  scenarios  independently,  F( 3,  159)  =  .418, 
p  =  .74,  partial  eta2  =  .008.  There  was  also  no  main  effect  of  experience  on  the  number  of 
details  searched,  F(l,  53)  =  .028.  p  =  .87,  partial  eta2  =  .001.  There  was  no  interaction  of 
experience  and  scenario  on  the  number  of  details  searched,  whether  scenarios  were  collapsed 
within  levels  of  certainty,  F(l,  57)  =  0.057,  p  =  .SI,  partied  eta2  =  .001,  or  treated  independently, 
F( 3,  159)  =  0.549,  p  =  .65,  partial  eta2  =  .010.  See  Table  3  for  the  descriptive  data  on  the 
number  of  searches  per  scenario. 


5  Soldiers  were  allowed  to  take  notes  while  reading  through  each  scenario,  but  it  is  unclear  whether  these  notes 
influenced  the  number  of  cues  identified  as  informative.  These  notes  are  available  for  analysis;  however,  because 
note  taking  was  not  systematic  (i.e.,  formal  instruction  was  not  given  on  how  to  take  notes,  nor  was  note-taking 
monitored),  they  have  not  been  coded. 
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Table  3 


Mean  Number  of  Information  Searches  Per  Scenario 


Scenario 

Experience  Level 

Inexperienced  Experienced 

Total 

Mean 

SD 

Mean 

SD 

Mean 

SD 

Dog  Day 

1.36 

1.29 

1.50 

1.48 

1.44 

1.39 

The  Rock 

1.72 

1.43 

1.63 

1.51 

1.67 

1.47 

Checkpoint 

1.44 

1.45 

1.62 

1.49 

1.54 

1.46 

Election 

1.80 

1.38 

1.48 

1.39 

1.61 

1.39 

Across  Scenarios 

1.58 

1.39 

1.56 

1.47 

1.57 

1.43 

One  possible  reason  Soldiers  conducted  so  few  searches  is  that  each  search  cost  the 
participant  $5  (virtual  dollars).  Soldiers  were  told  that  they  needed  to  complete  ten  scenarios, 
thus  they  had  to  conserve  money.  If  a  Soldier  calculated  the  amount  of  money  required  to 
complete  the  study,  this  calculation  would  reveal  that  an  average  of  two  details  per  scenario 
could  be  searched.  Moreover,  Soldiers  were  instructed  that  some  scenarios  would  be  quite 
difficult,  thus  perhaps  requiring  more  money.  The  mean  number  of  searches  observed  across 
scenarios  ( M  =  1.57)  might  support  the  notion  that  Soldiers  took  these  factors  (limited  resources 
and  anticipating  conserving  money  for  future  scenarios)  into  consideration.  Thus,  they  restricted 
their  searches  only  to  items  that  appeared  to  be  relevant  and  useful. 

Relevance  of  information  searched.  As  noted  in  the  section  on  manipulation  checks, 
both  experienced  and  inexperienced  Soldiers  tended  to  select  relevant  details  ( M  =  .80,  SD  =  .45) 
more  often  than  irrelevant  details  (M  =  .50,  SD  =  .38),  t(54)  =  3.95,  p  <  .001,  d  =  .82.  Proportion 
scores  were  calculated  to  determine  the  ratio  of  relevant  to  irrelevant  details  searched.  These 
proportion  scores  were  then  used  as  outcome  variables  to  analyze  the  effects  of  experience  and 
scenario  on  selection  of  relevant  details. 

There  was  no  effect  of  experience  on  selecting  relevant  versus  irrelevant  details 
( MExperienced  =  .72  vs.  M inexperienced  =  .77),  F(  1 ,  13)  =  .337,  p  =  .57,  partial  eta2  =  .025.  There  was 
also  no  effect  of  scenario  on  selecting  relevant  versus  irrelevant  details,  F( 3,  39)  =  .165 ,p  =  .92, 
partial  eta 2  =  .012.  There  was  no  interaction  of  experience  and  scenario  on  selecting  relevant 
versus  irrelevant  details,  F( 3,  39)  =  .302,  p  =  .82,  partial  eta2  =  .023.  A  constrained  sample  may 
have  limited  these  particular  analyses.  If  a  Soldier  did  not  search  any  details  in  one  scenario, 
Soldiers  who  did  not  search  any  details  in  one  or  more  scenarios  were  excluded  from  this 
analysis.  Five  experienced  and  nine  inexperienced  Soldiers  opted  not  to  search  any  details  in  at 
least  one  scenario.  This  restricted  sample  may  have  limited  the  power  in  this  analysis.  To 
overcome  data  lost  in  this  analysis,  we  conducted  pairwise  comparisons  based  on  experience  for 
each  scenario.  For  example,  a  Soldier  who  refrained  from  searching  details  on  The  Rock 
scenario,  but  searched  details  on  the  Checkpoint  scenario,  was  excluded  from  the  previous 
ANOVA,  but  was  included  in  a  between-subjects  analysis  of  the  Checkpoint  scenario.  These 
analyses  also  yielded  no  differences  (no  t  >  1.09,  no  p  <  .283). 
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Again,  the  cost  structure  imposed  on  Soldiers  in  these  tasks  may  have  influenced  their 
information  searching.  Because  resources  were  limited.  Soldiers  may  have  realized  that  they 
could  search  only  one  or  two  items  per  scenario,  and  thus,  regardless  of  their  experience  level, 
they  were  compelled  to  search  only  those  items  that  seemed  to  be  the  most  relevant.  A  lack  of 
difference  by  experience  level  and  scenario  may  also  have  been  a  result  of  the  way  in  which 
additional  details  were  constructed  and  labeled.  Items  may  have  been  obviously  relevant  or 
irrelevant  leading  to  ceiling  effects  for  selecting  relevant  items  and  floor  effects  for  selecting 
irrelevant  items.  Despite  this  possibility,  Soldiers  appeared  to  evaluate  the  scenarios  critically 
enough  to  compel  them  to  search  information  that  they  thought  should  be  relevant  to  their 
decisions. 

Confirmation  bias:  Association  between  initial  threat  decisions  and  new  details 
searched.  Some  additional  details  available  to  Soldiers  after  reading  through  each  scenario  were 
likely  to  be  similar  to  details  associated  with  identified  priority  threats.  To  understand  these 
associations,  we  matched  initial  threat  decisions  to  the  details  searched  after  reading  the 
scenarios.  These  data  were  analyzed  using  chi-square  (Table  4). 

Scenarios.  In  the  Dog  Day  scenario,  experienced  and  inexperienced  Soldiers  were 
equally  likely  to  confirm  their  initial  threat  hypotheses  by  searching  through  added  details  that 
matched  the  topic  of  their  identified  priority  threat,  y2  (1,  59)  =  .355,  p  =  .55.  In  The  Rock 
scenario,  experienced  and  inexperienced  Soldiers  were  equally  likely  to  confirm  their  initial 
hypotheses,/2  (1,  57)  =  1.436, p  =  .23.  In  the  Checkpoint  scenario,  again,  experienced  and 
inexperienced  Soldiers  were  equally  likely  to  confirm  their  initial  threat  hypotheses  by  searching 
through  added  details  that  matched  the  topic  of  their  identified  priority  threat,  y2  (1,  59)  =  0.213, 
p  =  .64.  In  the  Election  scenario,  experienced  Soldiers  and  inexperienced  Soldiers  were  equally 
likely  to  confirm  their  initial  hypotheses,  though  this  difference  reached  marginal  significance, 

Z2  (1,  59)  =  3.276,  p  =  .07. 

Table  4 

Percentage  of  Soldiers  Whose  Searches  Matched  Their  Initial  Threat  Decision 


Scenario 

Experienced 

Inexperienced  Experienced 

Z2 

P 

Dog  Day 

12 

21 

0.355 

0.55 

The  Rock 

20 

34 

1.436 

0.23 

Checkpoint 

24 

29 

0.213 

0.64 

Election 

24 

47 

3.276 

0.07 

Across  Scenarios 

20 

32.75 

Although  no  individual  chi-square  value  reached  statistical  significance,  experienced 
Soldiers  ostensibly  tended  to  search  information  that  matched  their  initial  hypotheses  more  often 
than  did  inexperienced  Soldiers.  To  test  this  trend  further,  we  calculated  a  likelihood  score  and 
analyzed  it  for  all  Soldiers.  The  dichotomous  variable  representing  confirmation  (“yes”  if  a 
Soldier  searched  a  matching  term  vs.  “no”  if  a  Soldier  did  not  search  a  matching  term)  was 
transformed  into  a  proportion  score  (number  of  confirmations/number  of  scenarios).  These 
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proportions  were  then  analyzed  with  an  independent  samples  t-test  which  revealed  that,  across  all 
scenarios,  experienced  Soldiers  tended  to  be  more  likely  than  inexperienced  Soldiers  to  search 
details  that  matched  their  initial  threat  decisions  ( M  =  .31,  SD  =  .22  vs.  M  =  .20,  SD  =  .22),  t( 57) 
=  2.06,  p  =  .04,  cl  =  .50.  When  collapsed  within  level  of  certainty,  there  was  no  main  effect  of 
‘certainty’  on  confirmation  likelihood,  t( 56)  =  1.56 ,p  =  .13,  d  =  0.29.  There  was  no  interaction 
of  experience  and  certainty  on  confirmation  likelihood,  F(l,  55)  =  0.06.  p  =  .82,  partied  eta2  = 
.001. 


While  it  is  possible  that  experienced  Soldiers  did  engage  in  confirmation  bias,  there  is 
one  alternative  explanation.  Experienced  Soldiers  may  have  generated  initial  hypotheses  that 
contained  information  similar  to  the  additional  details  presented,  thus  they  could  only  search 
similar  information.  Perhaps  their  hypotheses  were  more  relevant  and  more  readily  testable 
against  the  available  relevant  details.  Examining  initial  confidence  levels  might  clarify  these  two 
explanations.  For  example,  if  experienced  Soldiers  were  demonstrating  confirmation  bias,  we 
might  expect  confidence  in  their  initial  hypotheses  to  be  higher  than  that  of  inexperienced 
Soldiers  who  appeared  to  engage  in  confirmation  bias  to  a  lesser  extent. 

Initial  confidence.  Initial  confidence  ratings  were  analyzed  using  a  repeated  measures 
ANOVA  with  experience  as  the  between-subjects  factor  and  scenario  as  the  within-subjects 
factor.  These  ratings  failed  to  pass  Mauchly’s  test  of  sphericity6,  Mauch/y ’s  y2  (5)  =  12. 173,  p  = 
.033;  therefore,  subsequent  F  statistics  are  reported  using  the  Huynh-Feldt  correction 
(< Greenhouse-Geisser  Epsilon  >  .775).  Experienced  and  inexperienced  Soldiers  did  not  differ  in 
confidence  ratings  ( M Experienced  =  86.94,  SD  =  13.70  VS.  M Inexperienced  =  82.27,  SD  =  15.31),  /(51)  = 
1.17,  p  =  .248.  There  was  no  main  effect  of  scenario  on  initial  confidence  F(2.19,  142.37)  = 
0.736,  p  =  .52,  partied  eta 2  =  .014.  When  collapsed  within  level  of  certainty,  there  was  a  main 
effect  of  scenario  certainty  on  initial  confidence.  Initial  confidence  was  greater  in  certain 
scenarios  (M  =  86.1,  SD  =  14.6)  than  in  uncertain  scenarios  (M  =  81.6,  SD  =  17.3),  f(55)  =  2.563, 
p  =  .01,  d  =  0.28.  There  was  no  observed  interaction  between  experience  and  scenario  on 
confidence,  Greenhouse-Geisser  corrected  F(2.79,  142.37)  =  1 .00,  p  =  .39,  partial  eta2  =  .019. 

Overall,  both  experienced  and  inexperienced  Soldiers  tended  to  report  relatively  high 
confidence  in  their  initial  threat  decisions.  This  is  not  surprising  considering  they  were  rating 
their  own  hypotheses,  not  hypotheses  generated  by  others  (Adsit  &  London,  1997).  It  would  be 
unlikely  that  Soldiers  would  report  a  hypothesis  in  which  they  had  little  confidence.  This  notion 
is  supported  by  memory  research  demonstrating  that  people  tend  to  withhold  information  when 
the  accuracy  of  the  information  is  questionable  and  their  confidence  is  low  (Koriat  &  Goldsmith, 
1994).  When  potentially  unreliable  information  is  withheld,  reports  become  more  informative. 

In  this  experiment,  Soldiers  may  have  been  seeking  to  maximize  the  information  revealed  in 
their  hypotheses.  They  also  appeared  to  evaluate  these  scenarios  critically  and  this  perhaps 
resulted  in  a  judgment  of  learning  (Koriat,  1997)  in  which  the  process  of  reading  and  hearing  the 
scenario  provided  a  sense  of  having  learned  the  critical  features  of  the  scenario  which  translated 
into  confidence  in  recalling  and  reporting  those  critical  features.  The  previous  results  indicating 
that  certain  scenarios  tended  to  yield  more  identified  threats  and  informative  cues  than  did 
uncertain  scenarios  supports  this  notion.  If  the  ability  to  report  more  items  increased  a  sense  of 


6  See  Field  (2005)  for  Mauchly’s  test  of  sphericity  and  the  Huynh-Feldt  correction. 
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fluency  for  the  certain  scenarios,  then  it  would  be  reasonable  to  expect  that  confidence  would 
inflate  for  these  scenarios. 

Change  in  hypothesis.  Soldiers  changed  their  hypotheses  by  adding  information  that 
altered  the  initial  hypothesis  (e.g.,  the  priority  threat  was  changed  from  IEDs  to  spies)  or 
extended  the  initial  hypothesis.  Overall,  very  few  changes  in  hypotheses  occurred  (17%  [n  =  41] 
of  all  hypotheses  changed).  Multiple  logistic  regression  analyses  were  conducted  to  predict  a 
change  in  hypothesis  using  experience  and  initial  confidence  as  predictors.  In  the  Dog  Day 
scenario,  the  regression  was  not  statistically  significant,/2  (3)  =  6.13, p  =  .11,  indicating  that 
none  of  the  predictors  had  a  significant  effect  on  changes  in  hypothesis.  However,  initial 
confidence  was  a  marginally  significant  predictor  of  change  in  hypothesis  with  a  Wald  statistic7 
of  2.74,  p  =  .  10;  no  other  p  values  <  .703.  Changes  in  hypothesis  tended  to  occur  more 
frequently  with  lower  initial  confidence  ratings. 

In  The  Rock  scenario,  the  regression  was  statistically  significant,  /2  (3)  =  1 1.22,  p  =  .01. 
All  predictors  reached  statistical  significance:  confidence  attained  a  Wald  statistic  of  5.25,  p  = 
.02;  experience  attained  a  Wald  statistic  of  4.74,  p  =  .03;  and  the  interaction  of  initial  confidence 
and  experience  attained  a  Wald  statistic  of  4.74,  p  =  .03.  Inexperienced  Soldiers  who  changed 
their  hypotheses  had  the  same  confidence  levels  as  those  who  did  not  change  their  hypotheses. 

In  contrast,  experienced  Soldiers  who  changed  their  hypotheses  tended  to  report  lower  initial 
confidence  ratings.  The  interaction  is  presented  in  Figure  5. 

In  the  Checkpoint  scenario,  the  regression  was  not  statistically  significant,  /2  (3)  =  1.66,  p 
=  .65.  No  variables  predicted  change  in  hypothesis,  no  p  values  <  .578. 

In  the  Election  scenario,  the  regression  was  statistically  significant,/2  (3)  =  9.78, p  =  .02. 
Initial  confidence  was  the  closest  to  statistical  significance  with  a  Wald  statistic  of  2.47 ,  p  =  .12, 
thus  formally  not  significant.  Similar  to  the  trend  in  the  Dog  Day  scenario,  changes  in 
hypothesis  tended  to  occur  more  frequently  with  lower  initial  confidence  ratings. 

Changes  in  hypotheses  did  not  occur  frequently,  despite  Soldiers  having  access  to 
additional  information.  Soldiers  may  have  refrained  from  changing  their  hypotheses  because 
they  placed  greater  weight  on  the  initial  set  of  cues  compared  to  later  information.  This  may 
have  resulted  in  a  cue  primacy  effect  (Adelman,  Bresnick,  Black,  Marvin,  &  Sak,  1996).  Simply 
reporting  the  initial  cues,  which  were  informative,  may  have  increased  the  salience  of  these  cues 
during  later  evaluation  of  the  information.  Future  research  that  includes  a  protocol  in  which 
participants  do  not  report  initially  informative  cues  explicitly  might  help  elucidate  this  primacy 
effect.  It  may  be  informative  to  study  the  primacy  effect  in  a  real-world  threat  detection  context, 
as  it  suggests  that  making  informative  cues  explicitly  salient  may  lead  to  a  bias  against 
evaluating  any  novel  information  and  incorporating  it  into  a  revised  hypothesis. 


7  See  Field  (2005)  and  or  Menard  (2002)  for  information  on  Logistic  Regression,  including  the  Wald  statistic. 


18 


100 


Hypothesis  Change 


■  Experienced 

■  Inexperienced 


Figure  5.  Interaction  between  initial  confidence  and  experience  on  hypothesis  change  in  The 
Rock  scenario. 

Note:  Error  bars  represent  95%  CI. 


Overall,  changes  in  hypotheses  appear  to  be  associated  with  lower  initial  confidence 
ratings.  This  indicates  that  Soldiers  may  have  been  relatively  well  calibrated  (their  confidence 
and  accuracy  were  closely  related).  Soldiers  recognized  that  their  initial  hypotheses  may  have 
been  suboptimal  in  some  way,  and  confirmed  this  during  their  search  through  additional 
information.  Thus,  they  changed  their  hypothesis  to  reflect  their  new  knowledge  of  the  situation. 
Interestingly,  in  The  Rock  scenario,  inexperienced  Soldiers  appeared  poorly  calibrated.  Those 
who  changed  their  hypothesis  tended  to  be  more  confident  in  their  initial  hypotheses  than  were 
inexperienced  Soldiers  who  did  not  change  their  hypotheses.  These  results  should  be  evaluated 
with  caution.  Observed  changes  in  hypothesis  were  not  always  wholesale  changes,  instead 
additions  to  and  subtractions  from  hypotheses  were  scored  as  changes.  Thus,  Soldiers  may  have 
been  confident  in  their  initial  hypothesis.  They  may  have  added  a  detail  after  searching  through 
the  additional  information  that  did  not  substantively  change  the  hypothesis,  and  remained 
confident  in  that  initial  hypothesis. 

Conclusion  and  Future  Research 

A  few  trends  worth  further  consideration  emerged  from  this  experiment.  Experienced 
and  inexperienced  Soldiers  focused  on  different  priority  threats  across  three  of  the  four  scenarios, 
though  their  overall  patterns  of  responding  did  not  differ.  This  might  indicate  that  Soldiers  who 
previously  deployed,  and  went  outside  the  wire,  attend  to  different  cues  in  the  environment  and 
possibly  process  these  cues  differently.  Supporting  this  notion,  experienced  Soldiers  were  likely 
to  report  more  discrete  threats  when  identifying  their  priority  threat  in  each  scenario.  They  were 
also  more  likely  to  search  information  that  confirmed  their  initial  hypotheses.  The  finding  that 
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Soldiers’  initial  confidence  seemed  to  be  associated  with  their  inclination  to  change  their 
hypotheses  is  promising,  particularly  if  they  are  inclined  to  evaluate  and  revise  hypotheses  when 
they  are  not  maximally  confident.  Whether  or  not  Soldiers  changed  their  initial  hypotheses  is  a 
reasonable  measure  of  how  plausible  they  thought  their  hypotheses  were,  especially  given  the 
opportunity  to  evaluate  them  critically.  However,  future  research  should  incorporate  a  more 
direct  measure  of  the  confidence-accuracy  relationship  and  do  so  in  the  type  of  ill- structured 
decision  environment  we  used.  Traditional  calibration  studies  utilize  known  solutions  (at  least  in 
terms  of  using  probabilistically  derived  task  environments),  which  begets  only  a  finite  set  of 
responses  upon  which  calibration  can  be  assessed.  However,  the  operational  environment  is 
marked  by  dynamic  uncertainty.  As  such,  the  solutions  are  often  subjective  judgments  based  on 
subjective  cue  utilization  in  a  fluid  task  environment  with  shifting  instruction  sets  and  potential 
outcomes.  As  noted  by  Shanteau  (1989),  trying  to  understand  experts  or  the  development  of 
expertise  requires  an  understanding,  by  the  researcher,  that  experts  often  work  on  tasks  that  do 
not  always  have  correct  answers. 

Experienced  and  inexperienced  Soldiers  did  not  differ  on  other  measures,  such  as  the 
variability  in  their  priority  threats,  the  number  of  cues  identified  as  informative,  and  whether  they 
searched  relevant  additional  information  topics.  Perhaps  these  results  reflect  the  constraint 
imposed  by  the  testing  materials,  such  as  a  limited  number  of  cues  available  to  inform  the  initial 
hypothesis;  however,  they  may  also  reflect  general  trends  in  cognition.  Across  scenarios,  there 
was  little  difference  between  experienced  and  inexperienced  Soldiers  in  identifying  informative 
cues  (means  of  1.7  and  1.9  for  inexperienced  and  experienced  Soldiers,  respectively).  This 
finding  comports  with  literature  that  suggests  decision  makers  generate  hypotheses  using 
relatively  small  sets  of  data  (e.g.,  Klein,  1998).  Experience  did  not  influence  the  relevance  of 
information  searched;  moreover,  Soldiers  tended  to  search  relevant  details  more  often  than 
irrelevant  details.  As  with  the  association  between  initial  confidence  and  changes  in  hypotheses, 
this  finding  is  promising,  as  it  indicates  that  Soldiers  are  evaluating  their  hypothesis  based  on 
information  widely  considered  as  relevant.  Future  research  might  extend  beyond  evaluating 
threat  detection  and  assessment  to  include  action  choices,  for  instance,  through  more  interactive 
testing  in  which  participants  can  decide  on  and  implement  a  COA.  As  an  example,  future 
research  could  extend  these  findings  to  determine  if  the  information  Soldiers  deem  relevant  has 
any  influence  on  successful  courses  of  action  as  well  as  the  subsequent  consequences  and  further 
choices. 

Overall,  this  experiment  supports  the  idea  that  differences  exist  between  novice  and 
experienced  Soldiers’  decision  processes,  particularly  in  operationally  relevant  domains.  We  did 
not  find  that  inexperienced  Soldiers  would  base  their  hypotheses  on  a  wider  range  of  cues  than 
that  used  by  experienced  Soldiers.  We  also  failed  to  observe  differences  in  initial  confidence 
levels  associated  with  hypothesis  generation  and  the  extent  to  which  Soldiers  seek  additional  and 
relevant  information.  However,  we  discovered  that  (deployment)  experience  influences 
Soldiers’  tendency  to  confirm  their  hypotheses  and  that  initial  confidence  might  be  a  factor  that 
influences  changes  in  hypotheses.  Future  research  should  allow  testing  in  naturalistic 
environments  to  gain  a  greater  understanding  of  Soldier  decision  processes.  This  might  mitigate 
limitations  of  memory  inherent  in  computer-based  tests,  where  participants  only  passively 
experience  the  scenario  through  text  rather  than  a  comprehensive  experience.  For  example,  live 
scenarios  could  provide  a  larger,  richer  set  of  cues  available  for  informing  initial  hypotheses. 
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The  formation  and  testing  of  hypotheses  is  very  much  about  determining  potential  causes 
given  available  cues.  In  this  research,  we  examined  the  relationship  between  initial  confidence, 
hypothesis  generation  and  information  search  as  well  as  the  influence  of  the  certainty  of 
information  on  assessment  of  potentially  threatening  situations.  As  such,  the  focus  of  the 
analysis  was  on  the  decision  outcomes  or  products  rather  than  necessarily  on  the  external  or 
internal  factors  that  could  influence  cognitive  processing.  Future  research  could  involve 
exploring  the  factors  that  affect  the  formation  of  causal  links,  such  as  those  outlined  by  Einhom 
and  Hogarth  (1982),  including  contiguity  of  space,  similarity,  and  temporal  order  of  events 
within  a  decision  environment.  Likewise,  elements  of  predisposition  for  decision-making,  such 
as  decision-  making  style  or  approach  to  problems  (see  Driver,  1979;  Scott  &  Bruce,  1985)  could 
assist  in  determining  the  amount  of  information  persons  search  before  a  decision  is  reached  and, 
as  a  result,  assist  in  determining  the  primary  source  of  information  upon  which  hypotheses  were 
generated  and  the  decisions  were  made. 
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Appendix  A 


Scenarios  and  Questions 

Dog  Day 

You  are  assigned  to  Bravo  Co,  1/5  Cavalry  2nd  Platoon.  Your  Platoon  is  assigned  to  an  area  that 
has  seen  a  recent  upswing  in  violence  from  both  the  Taliban  and  growing  anger  at  the  U.S.  from 
perceived  insults  against  the  Islamic  religion  and  the  Qur  ’an.  Recent  Intel  indicates  that  the 
locals  have  held  several  meetings  where  members  discussed  ways  to  retaliate  against  the  U.S. 
You  and  your  Afghanistan  Army  counterparts  have  conducted  daily  patrols  in  your  sector  and 
have  experienced  an  increase  in  hostility,  but  have  not  experienced  an  increase  in  violence. 
During  a  recent  patrol,  you  arrested  several  locals  after  a  search  of  their  vehicle  uncovered 
several  components  for  an  IED.  During  questioning,  one  detainee  claimed  that  several  IEDs  are 
hidden  in  the  village,  but  he  would  not  provide  any  more  information. 

Your  S-2  was  able  to  provide  the  following  information  from  the  previous  unit  in  the  area. 
Working  with  the  locals  in  the  area  for  the  last  year,  the  previous  unit  seemed  to  be  winning  the 
hearts  and  minds  of  the  people.  They  worked  together  to  bring  a  potable  water  source  to  the 
village  and  they  formed  a  neighborhood  watch  made  up  of  the  local  civilian  population  who 
inform  the  unit  of  anything  disturbing  the  peace  in  the  area.  They  founded  a  school  and  helped 
deliver  school  supplies.  The  population  seemed  very  pleased  with  this  progress.  Several  months 
ago,  new  faces  started  appearing  in  the  area  and  the  local  people  became  less  friendly  and  more 
anti-U.S.  in  their  attitudes.  Two  months  before  your  unit  took  over  the  mission  in  the  area,  the 
other  unit  killed  four  of  the  newcomers  while  they  were  attempting  to  emplace  an  IED.  The  four 
people  killed  were  Pakistani  and  had  links  to  the  Taliban.  Intel  indicates  that  the  Taliban  have  a 
new  cleric  in  the  area  who  is  inflaming  anti-U.S.  tensions  with  the  local  population.  The  former 
unit  was  unable  to  identify  or  locate  the  cleric.  Prior  to  the  arrival  of  the  new  residents  and  the 
cleric,  overall  violence  in  the  area  was  declining  and  the  previous  unit  had  made  great  progress 
toward  turning  the  area  over  to  the  Afghan  forces  on  schedule.  You  suspect  that  some  of  the 
detainees  are  newcomers  who  have  been  disturbing  the  peace  but  no  one  has  yet  been  able  to 
confirm  this. 

You  are  now  out  on  patrol  with  trained  dogs  looking  for  the  IEDs  and  IED  components.  After 
several  searches  of  the  area,  the  dogs  keep  returning  to  the  same  location.  However,  you  do  not 
see  anything  unusual  or  out  of  place  in  that  location.  The  suspicious  location  is  in  front  of  a 
house  where  one  of  the  detainees  said  he  lives.  The  street  you  are  on  is  usually  full  of  activity. 
Today  the  street  is  relatively  quiet.  Two  women  are  doing  laundry  in  front  of  the  detainee ’s 
house  and  several  men  are  about  a  block  away,  doing  mechanical  work  on  a  pickup  truck. 

Instructions:  You  will  now  answer  a  series  of  questions  about  this  scenario.  First,  you  will 
describe  the  highest  priority  threat  in  this  scenario  and  explain  why  this  threat  is  important.  Next, 
you  will  answer  questions  concerning  your  confidence  and  the  cues  you  used  to  make  your 
assessment.  Following  this,  you  will  have  the  opportunity  to  search  for  more  information  about 
this  situation.  You  will  be  required  to  explore  at  least  one  piece  of  information  per  scenario. 
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Remember,  each  piece  of  information  you  gather  costs  you  money.  You  need  enough  money  to 
get  through  all  10  scenarios,  so  review  only  as  much  information  as  you  think  you  need  to  make 
a  good  assessment  of  the  threat.  Finally,  you  will  have  the  opportunity  to  revise  or  add  to  your 
original  assessment,  if  you  choose  to. 

1.  Describe  the  highest  priority  threat  in  this  scenario  and  explain  why  this  threat  is 

important. 

a.  Please  indicate  how  confident  you  are  that  this  threat  takes  the  highest  priority. 
Provide  a  percentage  from  0%  (no  confidence)  to  100%  (absolute  confidence). 

b.  List  the  information  or  cues  you  focused  on  to  identify  this  threat. 

2.  What  course  of  action  would  you  take  to  deal  with  this  threat? 

a.  What  outcome  do  you  expect  by  taking  this  course  of  action? 

b.  Please  indicate  how  confident  you  are  that  this  course  of  action  will  result  in  the 
anticipated  outcome.  Provide  a  percentage  from  0%  (no  confidence)  to  100% 
(absolute  confidence). 


Click  on  any  of  the  topics  below  to  gather  more  information.  You  must  select  at  least  one  topic. 

•  Previous  threat  activity  in  the  area:  In  this  region,  Soldiers  have  discovered  that  the 
locals  often  bury  bomb-making  materials  until  needed,  (relevant)* 

•  The  detainee’s  house:  The  detainee’s  house  is  located  behind  a  large  wall.  An  old 
wheelbarrow  sits  out  front.  The  detainee’s  uncle  owns  this  house;  villagers  have  indicated 
that  this  uncle  is  close  with  the  new  cleric  in  the  area,  (relevant) 

•  The  women  doing  laundry:  The  women  have  been  tending  to  their  laundry  for 
approximately  30  minutes.  They  have  been  talking  and  laughing  quietly,  and  are  not 
paying  much  attention  to  the  Soldiers’  activities  (irrelevant) 

•  Typical  street  activity:  This  street  is  one  of  many  that  lead  to  a  popular  market.  Foot  and 
vehicle  traffic  tend  to  be  light  except  during  key  travel  times  to  the  market,  (irrelevant) 

•  The  IED  components  found  in  the  vehicle:  The  components  for  the  IEDs  are  from 
Pakistan.  Soldiers  found  many  of  the  same  key  components  for  IEDs,  but  these  did  not 
include  all  the  components  necessary  to  complete  construction  of  an  IED  (relevant) 

•  Scheduled  turnover  to  Afghan  Forces:  The  previous  unit  worked  hard  to  prepare  the 
area  for  an  on-time  turnover  to  Afghan  Forces,  but  the  recent  tension  has  increased  doubt 
that  the  turnover  will  happen  as  scheduled,  (irrelevant) 

*These  labels  are  provided  for  the  reader,  but  were  not  revealed  to  participants. 


3.  Describe  the  highest  priority  threat  in  this  scenario  and  explain  why  this  threat  is  important.  If 
there  are  no  changes  to  your  initial  response,  just  type,  “No  Change.” 

4.  What  are  some  alternative  explanations  for  the  information  and  cues  you  focused  on  to  make 
your  threat  assessment  (for  example,  a  man  could  be  red  in  the  face  because  he  has  been 
exerting  himself  or  because  he  is  embarrassed)? 

5.  In  addition  to  your  initial  course  of  action  choice,  what  other  actions  might  work  to  reduce  or 
eliminate  this  threat? 
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The  Rock 


You  are  assigned  to  Bravo  Co,  1/5  Cavalry  2nd  Platoon.  You  have  been  working  with  the  Afghan 
Army  in  a  nearby  sector  for  months  and  have  achieved  great  progress  in  training  the  Afghan 
Army  to  take  over  the  sector  when  you  leave.  Last  night,  the  Afghan  Army  in  this  sector  was 
attacked  by  a  large  number  of  Taliban  on  a  hit  and  run  mission  just  outside  the  Afghan/U.S.  base. 
During  the  attack,  four  Afghan  Soldiers  and  several  Taliban  were  killed,  along  with  three  local 
civilians.  One  U.S.  Soldier  was  wounded.  The  wounded  U.S.  Soldier  reported  that  a  large  protest 
was  going  on  outside  the  compound  and  the  gunmen  emerged  from  the  crowd,  surprising  the 
guards  on  duty  at  the  base.  Your  Platoon  was  called  in  to  patrol  the  streets  and  find  the  Taliban 
terrorists. 

The  S-2  reports  that  approximately  30  gunmen  carrying  small  arms,  sniper  rifles  and  RPGs 
carried  out  the  attack.  The  gunmen  may  have  incited  frenzy  among  the  protestors  so  they  could 
attack  the  Afghan  Army  and  escape  amid  the  chaos.  The  gunmen  seemed  to  know  response 
procedures  and  the  number  and  positions  of  personnel  guarding  the  compound,  so  someone  in 
the  Afghan  forces  may  have  supported  this  attack.  The  speed  at  which  the  gunmen  carried  out 
their  attack  suggests  that  they  have  extensive  training  in  tactics.  One  of  the  gunmen  killed  was 
the  brother  of  an  Afghan  Soldier  working  inside  the  compound.  One  of  the  bystanders  killed  was 
the  son  of  the  village  chief.  S-2  suspects  that  the  gunmen  have  a  large  weapon  cache  somewhere 
in  the  area.  Your  ROE  states  that  you  are  authorized  to  use  deadly  force  to  protect  yourselves, 
your  unit,  or  U.S.  allies.  You  should  suspect  and  apprehend  all  military  age  personnel  ages  15  to 
50  and  bring  them  to  the  compound  for  eyewitness  identification  and  questioning.  You  are  to 
search  all  buildings  and  structures  for  the  weapon  cache  or  plans  used  for  the  attack.  You  will 
begin  by  searching  a  row  of  buildings  owned  by  a  village  merchant  who  may  have  some  family 
members  in  the  Afghan  Army. 

Instructions:  You  will  now  answer  a  series  of  questions  about  this  scenario.  First,  you  will 
describe  the  highest  priority  threat  in  this  scenario  and  explain  why  this  threat  is  important.  Next, 
you  will  answer  questions  concerning  your  confidence  and  the  cues  you  used  to  make  your 
assessment.  Following  this,  you  will  have  the  opportunity  to  search  for  more  information  about 
this  situation.  You  will  be  required  to  explore  at  least  one  piece  of  information  per  scenario. 
Remember,  each  piece  of  information  you  gather  costs  you  money.  You  need  enough  money  to 
get  through  all  10  scenarios,  so  review  only  as  much  information  as  you  think  you  need  to  make 
a  good  assessment  of  the  threat.  Finally,  you  will  have  the  opportunity  to  revise  or  add  to  your 
original  assessment,  if  you  choose  to. 

1.  Describe  the  highest  priority  threat  in  this  scenario  and  explain  why  this  threat  is 

important. 

a.  Please  indicate  how  confident  you  are  that  this  threat  takes  the  highest  priority. 

Provide  a  percentage  from  0%  (no  confidence)  to  100%  (absolute  confidence). 

b.  List  the  information  or  cues  you  focused  on  to  identify  this  threat. 

2.  What  course  of  action  would  you  take  to  deal  with  this  threat? 

a.  What  outcome  do  you  expect  by  taking  this  course  of  action? 
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b.  Please  indicate  how  confident  you  are  that  this  course  of  action  will  result  in  the 
anticipated  outcome.  Provide  a  percentage  from  0%  (no  confidence)  to  100% 
(absolute  confidence). 


Click  on  any  of  the  topics  below  to  gather  more  information.  You  must  select  at  least  one  topic. 

•  Known  connections  between  Afghan  Soldiers  and  attackers:  In  addition  to  the  brother 
of  an  Afghan  Soldier,  two  of  the  gunmen  killed  went  to  the  same  school  as  one  of  the 
Afghan  Soldiers  killed  in  the  attack,  (relevant) 

•  Location  where  the  Afghan  Soldiers  were  killed:  The  four  Afghan  Soldiers  were  killed 
near  the  front  gate  to  the  base,  (irrelevant) 

•  Possible  locations  of  the  gun  cache:  Within  the  past  two  months.  Soldiers  swept  several 
buildings  in  the  area  for  weapons  and  found  caches  of  small  arms.  Intel  indicates  a 
merchant  who  has  business  dealings  with  the  Taliban  owns  these  buildings,  (relevant) 

•  Background  on  the  son  of  the  village  chief:  The  son  of  the  village  chief  was  a  16-year 
old  student  who  worked  alongside  his  father  to  improve  living  conditions  in  the  village. 
He  was  active  in  anti-Taliban  efforts  in  the  community,  (irrelevant) 

•  The  reason  for  the  protest:  Villagers  were  protesting  American  use  of  drones  in  the 
area,  (relevant) 

•  The  number  of  military  age  men  in  the  village:  There  are  approximately  120-130 
military  age  men  in  this  village,  (irrelevant) 


3.  Describe  the  highest  priority  threat  in  this  scenario  and  explain  why  this  threat  is 
important.  If  there  are  no  changes  to  your  initial  response,  just  type,  “No  Change.” 

4.  What  are  some  alternative  explanations  for  the  information  and  cues  you  focused  on  to 
make  your  threat  assessment  (for  example,  a  man  could  be  red  in  the  face  because  he  has 
been  exerting  himself  or  because  he  is  embarrassed)? 

5.  In  addition  to  your  initial  course  of  action  choice,  what  other  actions  might  work  to 
reduce  or  eliminate  this  threat? 
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Checkpoint 


You  are  the  Platoon  leader  of  2nd  platoon,  B  Co  1/7  Cav.  and  you  have  just  been  assigned  to  this 
area.  The  outgoing  unit  briefed  you  on  the  area,  letting  you  know  that  violence  is  generally  low 
in  this  large  farming  community.  Any  violence  that  occurs  tends  to  come  from  outside  the  sector. 
The  previous  unit  kept  the  violence  low  by  conducting  constant  patrols  at  different  times  of  the 
day  and  by  developing  a  great  relationship  with  the  Elders  in  the  area.  The  Elders  seem  to  have 
a  strong  leadership  presence  in  the  community  and  have  established  laws  amongst  themselves,  as 
if  a  formal  government  does  not  exist.  While  out  on  patrol,  your  Bravo  section  setup  a  checkpoint 
to  look  for  Taliban  extremists  reportedly  heading  for  your  sector.  The  checkpoint  is  on  a  road 
commonly  used  by  local  farmers  to  transport  their  produce.  At  this  checkpoint,  they  detained  a 
brother  of  one  of  the  Elders  after  he  refused  a  body  search,  perceiving  it  as  an  insult.  The  Elders 
refuse  to  speak  with  you  or  any  U.S.  Soldier  until  you  release  the  Elder  s  brother  and  the  U.S. 
government  issues  a  formal  apology.  You  informed  your  chain  of  command  about  this  situation 
and  you  are  waiting  for  a  response.  Tensions  in  the  community  are  now  running  high  as  the  lack 
of  communication  between  your  forces  and  the  Elders  council  has  people  in  the  community 
refusing  to  speak.  You  have  heard  reports  that  some  younger  members  of  the  community  want  to 
take  revenge  for  the  perceived  insult.  Historically,  the  Elders  have  looked  down  on  these  types  of 
activities,  but  without  communication,  you  cannot  be  sure  of  their  stance  in  this  situation.  You 
have  closed  the  road  leading  to  and  from  the  front  of  your  compound  and  are  searching  anyone 
who  could  pose  a  threat  to  your  base. 

Intelligence  suggests  no  active  military  insurgents  in  this  area,  mainly  because  the  local  laws 
require  swift  punishment  of  insurgent  activities.  Previous  attacks  in  this  sector  were  tracked  to 
groups  coming  over  the  mountains  from  Pakistan  and  several  potential  attacks  have  been 
stopped  thanks  to  UAV patrols  along  the  border.  Intel  indicates  that  the  locals  own  only  small 
arms,  which  they  use  to  protect  their  homes  and  farms.  It  is  critical  for  U.S.  troops  to  keep  on 
good  terms  with  the  Elder  council  because  the  information  they  provide  has  been  vital  to 
knowing  Taliban  movement  in  the  area. 

Instructions:  You  will  now  answer  a  series  of  questions  about  this  scenario.  First,  you  will 
describe  the  highest  priority  threat  in  this  scenario  and  explain  why  this  threat  is  important.  Next, 
you  will  answer  questions  concerning  your  confidence  and  the  cues  you  used  to  make  your 
assessment.  Following  this,  you  will  have  the  opportunity  to  search  for  more  information  about 
this  situation.  You  will  be  required  to  explore  at  least  one  piece  of  information  per  scenario. 
Remember,  each  piece  of  information  you  gather  costs  you  money.  You  need  enough  money  to 
get  through  all  10  scenarios,  so  review  only  as  much  information  as  you  think  you  need  to  make 
a  good  assessment  of  the  threat.  Finally,  you  will  have  the  opportunity  to  revise  or  add  to  your 
original  assessment,  if  you  choose  to. 

1.  Describe  the  highest  priority  threat  in  this  scenario  and  explain  why  this  threat  is 

important. 

a.  Please  indicate  how  confident  you  are  that  this  threat  takes  the  highest  priority. 

Provide  a  percentage  from  0%  (no  confidence)  to  100%  (absolute  confidence). 

b.  List  the  information  or  cues  you  focused  on  to  identify  this  threat. 
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2.  What  course  of  action  would  you  take  to  deal  with  this  threat? 

a.  What  outcome  do  you  expect  by  taking  this  course  of  action? 

b.  Please  indicate  how  confident  you  are  that  this  course  of  action  will  result  in  the 
anticipated  outcome.  Provide  a  percentage  from  0%  (no  confidence)  to  100% 
(absolute  confidence). 


Click  on  any  of  the  topics  below  to  gather  more  information.  You  must  select  at  least  one  topic. 

•  Recent  activities  of  young  males  in  the  community:  The  younger  males  in  the  area 
have  been  very  vocal  about  the  arrest  and  have  staged  several  small  protests.  There  are 
rumors  that  they  have  held  a  series  of  closed-door  meetings  without  including  the  Elders, 
(relevant) 

•  Farming  conditions:  The  weather  and  time  of  year  are  great  for  harvesting  the  local 
grain.  More  farmers  have  been  delivering  goods  to  the  market  place  in  recent  weeks, 
(irrelevant) 

•  The  Elder’s  brother:  There  is  no  existing  Intel  on  the  brother  and  he  has  no  known  ties 
to  the  Taliban.  Troops  found  a  small  amount  of  drugs  on  the  brother  when  they  searched 
him.  (relevant) 

•  Previous  peacekeeping  activities:  Troops  in  the  previous  unit  helped  the  community 
build  a  school  and  they  rebuilt  several  roads  around  the  market  place,  (irrelevant) 

•  Recent  Elder  activity:  The  Elders  recently  cancelled  a  meeting  with  you  and  told 
members  of  the  community  to  stop  working  for  the  U.S.,  leaving  many  without  a  source 
of  income,  (relevant) 

•  Laws  established  by  the  Elders:  The  laws  established  by  the  Elders  aim  to  keep 
violence  down  and  maintain  order  in  the  community.  The  Elders  impose  severe 
punishment  on  law-breakers  and  those  who  threaten  to  disrupt  the  peace  in  any  way. 
(irrelevant) 


3.  Describe  the  highest  priority  threat  in  this  scenario  and  explain  why  this  threat  is 
important.  If  there  are  no  changes  to  your  initial  response,  just  type,  “No  Change.” 

4.  What  are  some  alternative  explanations  for  the  information  and  cues  you  focused  on  to 
make  your  threat  assessment  (for  example,  a  man  could  be  red  in  the  face  because  he  has 
been  exerting  himself  or  because  he  is  embarrassed)? 

5.  In  addition  to  your  initial  course  of  action  choice,  what  other  actions  might  work  to 
reduce  or  eliminate  this  threat? 
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Election 


You  are  assigned  to  3rd  Platoon,  Bravo  Company  1/5  Cav.  Your  Platoon  has  been  assigned  to 
support  the  Coalition  forces  in  protecting  the  civilian  population  as  they  gather  to  cast  their  vote 
in  the  region ’s  first  free  election.  Because  the  residents  in  this  region  openly  support  the 
Coalition  and  free  elections,  Coalition  forces  fear  that  the  Taliban  and  insurgent  forces  will 
attempt  to  disrupt  the  elections.  This  area  has  been  free  of  attacks  for  severed  months  and  the 
area  remains  stable,  largely  because  the  region  ’s  Elders  worked  extensively  with  U.S.  and 
Coedition  forces  to  improve  conditions  and  increase  employment  opportunities  in  the  community. 
The  Elelers  einel  community  members  set  up  ein  election  heeielquarters  in  a  buileling  close  to  the 
centred  market  einel  they  heive  been  actively  promoting  the  elections  through  a  series  of 
community  meetings.  The  Elelers  heive  not  received  any  threats,  but  they  are  concerned  eibout 
security  during  the  elections.  Intel  sources  have  investigated  and  confirmed  that  no  creelible 
threats  currently  exist.  They  continue  to  monitor  the  situation.  The  Clerics  in  this  region 
supported  the  Coalition  until  recently  and  are  now  withdrawing  their  support  of  the  elections 
because  they  believe  that  women  should  not  vote,  in  accordance  with  the  Muslim  religion. 
Community  members  will  have  three  days  to  cast  their  vote  at  the  election  headquarters.  Your 
platoon  will  escort  local  vehicles  that  will  transport  voters  to  and  from  their  homes.  You  will 
also  be  part  of  the  standing  guard  rotation  at  the  election  headquarters. 

Your  S-2  has  no  knowledge  of  an  imminent  attack  and  there  are  no  alerts  of  immediate  danger. 
According  to  local  informants,  some  people  may  wish  to  disrupt  the  election  process,  perhaps  by 
preventing  voters  from  reaching  the  election  site,  but  it  is  not  clear  how  they  might  do  that. 

Instructions:  You  will  now  answer  a  series  of  questions  about  this  scenario.  First,  you  will 
describe  the  highest  priority  threat  in  this  scenario  and  explain  why  this  threat  is  important.  Next, 
you  will  answer  questions  concerning  your  confidence  and  the  cues  you  used  to  make  your 
assessment.  Following  this,  you  will  have  the  opportunity  to  search  for  more  information  about 
this  situation.  You  will  be  required  to  explore  at  least  one  piece  of  information  per  scenario. 
Remember,  each  piece  of  information  you  gather  costs  you  money.  You  need  enough  money  to 
get  through  all  10  scenarios,  so  review  only  as  much  information  as  you  think  you  need  to  make 
a  good  assessment  of  the  threat.  Finally,  you  will  have  the  opportunity  to  revise  or  add  to  your 
original  assessment,  if  you  choose  to. 

1.  Describe  the  highest  priority  threat  in  this  scenario  and  explain  why  this  threat  is 

important. 

a.  Please  indicate  how  confident  you  are  that  this  threat  takes  the  highest  priority. 
Provide  a  percentage  from  0%  (no  confidence)  to  100%  (absolute  confidence). 

b.  List  the  information  or  cues  you  focused  on  to  identify  this  threat. 

2.  What  course  of  action  would  you  take  to  deal  with  this  threat? 

a.  What  outcome  do  you  expect  by  taking  this  course  of  action? 

b.  Please  indicate  how  confident  you  are  that  this  course  of  action  will  result  in  the 
anticipated  outcome.  Provide  a  percentage  from  0%  (no  confidence)  to  100% 
(absolute  confidence). 
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Click  on  any  of  the  topics  below  to  gather  more  information.  You  must  select  at  least  one  topic. 


•  Routes  to  the  election  headquarters:  The  voter  transportation  vehicles  will  pick  voters 
up  at  several  designated  meeting  places,  such  as  local  schools  and  mosques  and  drive 
them  through  the  central  market  to  the  headquarters,  (relevant) 

•  Elders’  election  promotion  activities:  The  Elders  held  a  series  of  publicized  meetings  to 
discuss  the  candidates  and  issues,  and  to  review  voting  procedures,  (irrelevant) 

•  Voter  activity  around  the  election  headquarters:  The  vehicles  drop  off  large  numbers 
of  voters  at  the  same  time.  This  causes  large  crowds  to  gather  around  the  front  door  as 
voters  wait  to  show  their  identification  and  go  inside,  (relevant) 

•  Voter  transportation  vehicles:  Voters  will  be  transported  in  buses  and  the  backs  of 
pickup  trucks  to  the  election  headquarters,  (irrelevant) 

•  Relationship  between  Clerics  and  Coalition:  The  Clerics  have  started  reaching  out  to 
the  community  to  gather  support  for  their  position  against  women  voting.  They  have  also 
ventured  outside  the  immediate  community  to  rally  support  for  their  position  on  women 
voters,  (relevant) 

•  Previous  use  of  the  election  headquarters  building:  The  election  headquarters  is 
located  in  a  building  that  was  used  as  a  storehouse  for  market  merchandise  until  a  larger 
building  was  built  on  the  other  side  of  the  central  market.  The  building  was  vacant  for 
several  years  before  the  Elders  decided  to  use  it  for  the  election,  (irrelevant) 


3.  Describe  the  highest  priority  threat  in  this  scenario  and  explain  why  this  threat  is 
important.  If  there  are  no  changes  to  your  initial  response,  just  type,  “No  Change.” 

4.  What  are  some  alternative  explanations  for  the  information  and  cues  you  focused  on  to 
make  your  threat  assessment  (for  example,  a  man  could  be  red  in  the  face  because  he  has 
been  exerting  himself  or  because  he  is  embarrassed)? 

5.  In  addition  to  your  initial  course  of  action  choice,  what  other  actions  might  work  to 
reduce  or  eliminate  this  threat? 
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Appendix  B 


Demographic  Questions 


1.  Time  in  service  (yrs) 

2.  Current  rank 

3.  Time  in  current  rank  (months) 

4.  Current  MOS 

5.  Age 

6.  Have  you  ever  deployed?  Yes  No 

7.  If  yes,  how  many  times  have  you  deployed? 

8.  Location  of  most  recent  deployment  (city  or  cities  and  country) 

9.  MOS  while  on  your  most  recent  deployment 

10.  How  often  did  you  go  outside  the  wire  on  your  most  recent  deployment? 

Never 

Less  than  once  a  month 

Once  a  month 

More  than  once  a  month 

Once  a  week 

More  than  once  a  week 

Everyday 

11.  Describe  some  of  your  duties  during  your  most  recent  deployment. 

12.  Describe  any  training  you  have  received  that  improved  your  ability  to  detect  threats. 
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